National Institute for Learning Outcomes Assessment |

National Institute for Learning Outcomes Assessment

NILOA Guest Viewpoints

We’ve invited learning outcomes experts and thought leaders to craft a Viewpoint. We hope that these pieces will spark further conversations and actions that help advance the field. To join the conversation, click the link below the Viewpoint. You can also sign up here to receive monthly newsletters that headline these pieces along with NILOA updates, current news items, and upcoming conferences.

 

Taking Stock of the Assessment Movement –
Liberal Education, Winter, 2017

Peter Ewell, Pat Hutchings, Jillian Kinzie, George Kuh & Paul Lingenfelter
NILOA Senior Scholars

 

Thanks to AAC&U for featuring the assessment movement in higher education in the Winter, 2017 issue of Liberal Education. The focus is timely, not only because it’s useful after 30 years of work to step back and take stock, but because the need is greater than ever that an even larger proportion of the nation’s citizens engage in high-quality postsecondary study.  Sound assessment of student learning is required to assure that colleges and universities have evidence to guide their continuous improvement efforts.

The National Institute for Learning Outcomes Assessment (NILOA) was established in 2008 to “discover and disseminate ways that academic programs and institutions can productively use assessment data internally to inform and strengthen undergraduate education….” Although NILOA is a logical outgrowth of the assessment movement that began decades earlier, its work is a direct response to the 2006 report, A Test of Leadership:  Charting the Future of American Higher Education, released by US Secretary of Education Margaret Spellings’ Commission on the Future of Higher Education. The Commission called for changing higher education “from a system primarily based on reputation to one based on performance.”

Many in the higher education community objected to particular recommendations and the “tone” of the Commission report, which seemed to reflect little confidence in academic leadership and undue confidence in the effectiveness of consumer information and hard-edged accountability measures.  At the same time, few disputed the growing importance of higher education and the need to increase the quality and extent of educational attainment in the United States. NILOA was founded to advance that purpose by improving assessment practices and by disseminating our growing knowledge about what works to promote student learning in undergraduate study.

Last spring NILOA issued a policy statement that summarizes what we have learned about how to approach the assessment of learning and using its results to obtain higher achievement. Indeed, using evidence of student learning for improvement is perhaps the strongest argument against excessive compliance requirements or crude accountability mechanisms. http://www.learningoutcomesassessment.org/NILOA_statement.html

In this context, the first three essays in the Winter, 2017 issue of Liberal Education are instructive for reviewing the challenges facing higher education, the achievements of the assessment movement, and the work that remains.

The opening essay by W. Russell Neuman, “Charting the Future of US Higher Education: A Look at the Spellings Report Ten Years Later,” focused on two facets of the “performance” of higher education which have competed for attention in the last decade: preparing more students for successful careers, and improving the quality of student learning outcomes and the number of students who succeed.

Career success obviously depends on what students know and can do, but it is not a simple matter to measure either of these outcomes. Neuman chronicles the academic community’s resistance to efforts to hold higher education accountable for performance based on crude outcomes like graduation rates and the pay and employment status of graduates. He also describes the limited successes of the community’s efforts to measure critical thinking and other learning outcomes through standardized measures like the Collegiate Learning Assessment (CLA), as well as a variety of evidence-gathering approaches advanced by the accreditation community.

Neuman argues that such efforts to focus on performance have gained some purchase at the institutional level, but they “seldom make themselves felt in classrooms and faculty offices.” In the next ten years, he suggests that technological innovations in teaching will naturally lead to a greater faculty focus on the intended educational outcomes of instruction and the relative effectiveness of different instructional approaches. He also suggests that technology is providing useful opportunities to enhance both assessment and instruction because it enables detailed real-time analysis of the student’s interactions with computer based instructional materials.

The second essay in the issue, “Toward an Improvement Paradigm for Academic Quality” by Douglas D. Roscoe, argues that an “improvement paradigm” should replace the “assessment paradigm" to assure quality in higher education.  Roscoe argues that assessment as currently practiced, mostly in response to institutional accreditation, is costly and increasingly ritualistic, which more often generates faculty resistance than it inspires improvement. In his words, “The reality is that the improvement piece of the assessment paradigm often takes a back seat to the collection of data.”

This is a serious concern. Over-emphasizing accountability and compliance naturally leads to a demand for precision of measurement that is impossible to achieve. Indeed, the most “reliable” and “valid” assessments, if they take the form of standardized tests, usually provide poor representations of the complex abilities that graduates should take away from a program of undergraduate study.

Roscoe’s proposed solution to this problem, however, is to focus on improving instruction directly through faculty conversation and the research literature without reference to a shared framework for assessing learning outcomes or, indeed, by collecting any new evidence at all. While research does provide helpful guidance for improvement, efforts to improve that do not start by defining the objectives of instruction are akin to sailing without a compass. We agree that “improvement” must be the desired end. But efforts to improve without shared instructional goals and associated measurements will not lead to systematic improvement.    

Consistent with this view, Terrell Rhodes’ essay, “The VALUE of Learning: Meaningful Assessment on the Rise,” chronicles the achievements of the assessment movement through the work of AAC&U. Rhodes is convinced that the academic community has made significant progress in articulating core learning objectives, developing meaningful, authentic assessments of learning, and improving instruction and attainment through use of the resulting information. As a case in point, he describes how AAC&U’s Valid Assessment of Undergraduate Education (VALUE) initiative, and its associated rubrics that address numerous collegiate proficiencies, is based on a broad consensus among educators and employers on desired learning outcomes, and emphasizes the importance of basing assessment on actual student work.

The utility of this approach has been widely demonstrated, perhaps most visibly by the Multi-State Collaborative to Advance Learning Outcomes Assessment (MSC) project undertaken jointly by AAC&U and the State Higher Education Executive Officers (SHEEO). Now in its third year, the MSC involves thirteen states using VALUE rubrics to rate tens of thousands of student artifacts in communications, quantitative reasoning, and critical thinking. NILOA has supported the work of the MSC in providing states with assignment design help in the form of state-wide charrettes. Meanwhile, NILOA’s own work using the Degree Qualifications Profile (DQP) is adding to this knowledge base by assisting about 800 institutions in harnessing the DQP and its associated Assignment Library, which contains a growing number of model assignments designed explicitly to yield scorable student artifacts for various DQP proficiencies. Finally, NILOA continues to assist institutional good practice in assessment through its periodic surveys of institutional assessment practices (the third of which will be out in the field in April 2017), its numerous publications on assessment, and its many web-based resources. Further, the NILOA 2015 book, Using Evidence of Student Learning to Improve Higher Education, focused upon moving the conversation on assessment away from a compliance/accountability exercise and towards more meaningful use, populated with various examples of how such work is unfolding in institutions throughout the US.

In sum, we are grateful to AAC&U for helping the field “take stock” of the assessment movement in the most recent issue of Liberal Education. We look forward to continued partnering with AAC&U and other organizations and individuals committed to furthering this important work in the years to come. As we conclude in our policy statement, “it is no longer beyond the capacity of a college or university to articulate expectations for learning, to document student progress toward these expectations, and to use the resulting evidence to improve student success...doing this job, and doing it well, is within our grasp…failing to do so shortchanges our students and the many others who have a major stake in the quality of higher education.”

 

Check out our past Viewpoints:

Taking Stock of the Assessment Movement – Liberal Education, Winter, 2017
Peter Ewell, Pat Hutchings, Jillian Kinzie, George Kuh & Paul Lingenfelter

Eight Years On: Early—and Continuing—Lessons from the Tuning Project
Daniel J. McInerney

Real-time Student Assessment: Prioritizing Enrolled Students’ Equitable Progress toward Achieving a High-Quality Degree
Peggy Maki

Academic and Student Affairs Sides of the House: Can We Have an Open Concept Learning Design?
Darby Roberts

Just Assessment. Nothing More. Nothing Less.
Wayne Jacobson

Design for a Transparent and Engaging Assessment Website
Frederick Burrack and Chris Urban

Improvement Matters
Peter Felten

Working Together to Define and Measure Learning in the Disciplines
Amanda Cook, Richard Arum, and Josipa Roksa

The Simplicity of Cycles
Mary Catharine Lennon

Helping Faculty Use Assessment Data to Provide More Equitable Learning Experiences
Mary-Ann Winkelmes

Ignorance is Not Bliss: Implementation Fidelity and Learning Improvement
Sara J. Finney and Kristen L. Smith

Student Learning Outcomes Alignment through Academic and Student Affairs Partnerships
Susan Platt and Sharlene Sayegh

The Transformation of Higher Education in America: Understanding the Changing Landscape
Michael Bassis

Learning-Oriented Assessment in Practice
David Carless

Moving Beyond Anarchy to Build a New Field
Hamish Coats

The Tools of Intentional Colleges and Universities: The DQP, ELOs, and Tuning
Paul L. Gaston, Trustees Professor, Kent State University

Addressing Assessment Fatigue by Keeping the Focus on Learning
George Kuh and Pat Hutchings, NILOA

Evidence of Student Learning: What Counts and What Matters for Improvement
Pat Hutchings, Jillian Kinzie, and George D. Kuh, NILOA

Using Evidence to Make a Difference
Stan Ikenberry and George Kuh, NILOA

Assessment - More than Numbers
Sheri Barrett

Challenges and Opportunities in Assessing the Capstone Experience in Australia
Nicolette Lee

Making Assessment Count
Maggie Bailey

Some Thoughts on Assessing Intercultural Competence
Darla K. Deardorff

Catalyst for Learning: ePortfolio-Based Outcomes Assessment
Laura M. Gambino and Bret Eynon

The Interstate Passport: A New Framework for Transfer
Peter Quigley, Patricia Shea, and Robert Turner

College Ratings: What Lessons Can We Learn from Other Sectors?
Nicholas Hillman

Guidelines to Consider in Being Strategic about Assessment
Larry A. Braskamp and Mark E. Engberg

An "Uncommon" View of the Common Core
Paul L. Gaston

Involving Undergraduates in Assessment: Documenting Student Engagement in Flipped Classrooms
Adriana Signorini & Robert Oschner

The Surprisingly Useful Practice of Meta-Assessment
Keston H. Fulcher & Megan Rodgers Good

Student Invovlement in Assessment: A 3-Way Win
Josie Welsh

Internships: Fertile Ground for Cultivating Integrative Learning
Alan W. Grose

What if the VSA Morphed into the VST?
George Kuh

Where is Culture in Higher Education Assessment and Evaluation?
Nora Gannon-Slater, Stafford Hood, and Thomas Schwandt

Embedded Assessment and Evidence-Based Curriculum Mapping: The Promise of Learning Analytics
Jane M. Souza

The DQP and the Creation of the Transformative Education Program at St. Augustine University
St. Augustine University

Why Student Learning Outcomes Assessment is Key to the Future of MOOCs

Wallace Boston & Jennifer Stephens

Measuring Success in Internationalization: What are Students Learning?
Madeleine F. Green

Demonstrating How Career Services Contribute to Student Learning
Julia Panke Makela & Gail S. Rooney

The Culture Change Imperative for Learning Assessment
Richard H. Hersh & Richard P. Keeling

Comments on the Commentaries about "Seven Red Herrings"
Roger Benjamin

Ethics and Assessment: When the Test is Life Itself
Edward L. Queen

Discussing the Data, Making Meaning of the Results
Anne Goodsell Love

Faculty Concerns About Student Learning Outcomes Assessment
Janet Fontenot

What to Consider When Selecting an Assessment Management System
R. Stephen RiCharde

AAHE Principles of Good Practice: Aging Nicely A Letter from Pat Hutchings, Peter Ewell, and Trudy Banta

The State of Assessment of Learning Outcomes Eduardo M. Ochoa

What is Satisfactory Performance? Measuring Students and Measuring Programs with Rubrics
Patricia DeWitt

Being Confident about Results from Rubrics Thomas P. Judd, Charles Secolsky & Clayton Allen

What Assessment Personnel Need to Know About IRBs
Curtis R. Naser

How Assessment and Institutional Research Staff Can Help Faculty with Student Learning Outcomes Assessment
Laura Blasi

Why Assess Student Learning? What the Measuring Stick Series Revealed
Gloria F. Shenoy

Putting Myself to the Test
Ama Nyamekye

From Uniformity to Personalization: How to Get the Most Out of Assessment
Peter Stokes

Transparency Drives Learning at Rio Salado College
Vernon Smith

Navigating a Perfect Storm
Robert Connor

It is Time to Make our Academic Standards Clear
Paul E. Lingenfelter

In Search for Standard of Quality
Michael Bassis

Avoiding a Tragedy of the Commons in Postsecondary Education
Roger Benjamin