National Institute for Learning Outcomes Assessment |

National Institute for Learning Outcomes Assessment

NILOA Guest Viewpoints

We’ve invited learning outcomes experts and thought leaders to craft a Viewpoint. We hope that these pieces will spark further conversations and actions that help advance the field. To join the conversation, click the link below the Viewpoint. You can also sign up here to receive monthly newsletters that headline these pieces along with NILOA updates, current news items, and upcoming conferences.

 

Real-time Student Assessment: Prioritizing Enrolled Students’ Equitable Progress toward Achieving a High-Quality Degree
Peggy Maki

 

There are pressing 21st-century reasons why continuously assessing enrolled students’ equitable progress toward achieving a high-quality degree needs to become the assessment norm across our colleges and universities. These reasons include:

  • Continuing diversification of our student demographics (representative of our democracy), representing broad ranges of academic preparation and readiness and personal needs;
  • Persistent gaps in achievement and degree completion rates between historically represented and historically underrepresented students;
  • The nation’s dependence on an educated citizenry from across our student demographics to ensure our prosperity, long-term growth, and attentiveness to our democratic values; and
  • High demands for associate- and bachelor-degreed students with relevant 21st- century skills to address evolving needs of the workplace and challenges of globalization.

Real-time Student Assessment: Meeting the Imperative for Improved Time to Degree, Closing the Opportunity Gap, and Assuring Student Competencies for 21st-Century Needs challenges institutions and their programs to prioritize the use of chronological assessment results to benefit enrolled students in comparison with the more common practices of prolonged assessment cycles or scheduled point-in-time practices, the results of which are generally used to benefit future students. To ensure that all students across our increasingly diverse student bodies graduate with the outcomes they need to be active and productive citizens, I advocate for real-time student assessment processes at the institution and program levels. Real-time student assessment continuously identifies and addresses patterns of student underperformance or obstacles that require timely interventions for enrolled students.  It is an on-the-ground, non-stop, living commitment to students’ equitable progress towards achieving a high-quality 21st-century degree. And, it is internally driven by institutional leadership and an inclusive commitment to all students’ success that engages faculty, administrators, and the institution’s extended network of experts who also contribute to and support student learning, such as professionals in student and academic support services or education technology.   

Real-time student assessment is not uncharted territory. Programs such as music, theater, art, and professional preparatory programs such as teacher education and nursing, for example, have long integrated this continuous assessment approach into their curricula. The foundation of competency-based education (CBE) programs and degrees also rests on chronologically assessing students’ performance toward mastery-level achievement. A small set of institutions, notably exemplified by Alverno College in its “Student Assessment-as-Learning” process, continuously assesses their students’ learning (Alverno, n.d.). In recent years, several large-scale higher education consortia have been formed, such as the University Innovation Alliance (UIA), consisting of 11 large research universities focused on closing achievement gaps and graduation rates among students from all social and ethnic backgrounds. Besides recognizing the need to reform conventional educational practices to better serve students, these institutions also recognize the need to remain informed about and respond on time to students’ academic and often personal needs (Vision and Prospectus, n.d.). What drives assessment in all of these examples is an internal need to know continuously how well all students are progressing toward achieving agreed-upon high-quality exit-level program and degree outcomes. They represent the assessment commitment that is now necessary across all of our institutions: an efficacious learner-centered assessment process that operates in the present tense, is internally motivated and driven, and values the nimble use of assessment results to advance all enrolled students to achieve a high-quality degree. However, these examples do not represent the assessment norm across our colleges and universities. Typically, institutions and programs take snapshots of persisting students’ performance during scheduled assessment cycles or at identified points in time. Typically, as well, interrogation of assessment results benefits future students more than it benefits enrolled students. In contrast the immediate beneficiaries of real-time student assessment are enrolled students. Real-time student assessment continuously gauges all enrolled students’ course-by-course and education experience-by-education experience progress from point of matriculation, transfer, or re-entry to point of graduation.

Real-time Student Assessment assists institutions and their programs in taking this continuous process to scale. This commitment is now more possible than ever before in higher education because of the following 21st-century grassroots developments grounded in purposeful engagement of faculty and other contributors to student learning across all sectors of higher education:

  • Lumina Foundation’s support of the development of: (a) the Degree Qualifications Profile that specifies what all undergraduates—regardless of their major programs of study—should know and be able to do and (b) Tuning, a collaborative process involving faculty; experts in a field, discipline, or profession; employers; and recent graduates that is focused on identifying the knowledge and skills students in specific disciplines, professions, or fields of study must demonstrate upon degree completion to transition successfully into their careers.
  • In addition, in recent years the Department of Education has become more open to receiving and approving Competency-Based Education (CBE) programs and degrees, particularly innovative ones that meet the needs of working students (Federal Student Aid Office, 2013).

Transcending a specific institutional type or mission or even students’ diverse pathways to a degree, these outcomes-based frameworks now provide institutions the continuous means to monitor students’ equitable progress to a high-quality degree—even if a pathway includes credit for prior learning, transfer of credits, or credentialed coursework from an external provider.

Anchored in these outcomes-based frameworks, Real-time Student Assessment identifies and discusses the five core learner-centered commitments that stretch across an agreed upon framework and altogether build a platform to support real-time student assessment. The canvas of these interdependent commitments makes it possible for faculty and other contributors to student learning to gauge continuously students’ equitable progress towards achieving high-quality institution- and program-level outcomes. For students, this canvas also charts a clear and coherent degree pathway, assisting them to understand the relevance of their coursework and other educational experiences and to gauge their time to degree. The canvas of five core learner-centered commitments includes the following:

Five Core Learner-centered Commitments that Stretch across A Shared Outcomes-based Framework

Commitment 1: Shared commitment to close measurably existing achievement and graduation gaps

Commitment 2: Agreement on the language of outcomes and aligned scoring rubrics to identify students’ patterns of performance and underperformance continuously

Commitment 3: Coherence across the curriculum, co-curriculum, and other educational experiences

Commitment 4: Alignment of courses, educational experiences, and assignments with outcomes and standards and criteria of judgment

Commitment 5: Faculty collaboration with the institution’s network of experts that includes, for example, professionals in academic and support services, education technology, faculty development, tutoring, advising, and mentoring.

Upon this platform, Real-time Student Assessment identifies, discusses, and illustrates through case studies how continuous interrogation and nimble use of assessment results occurs semester-by-semester (sometimes even mid-semester). This non-stop process closes existing time gaps that may exist in between an institution’s or its programs’ current assessment cycles or scheduled points-in-time approach to assessment. Why close those gaps? Because across those gaps faculty and other contributors to student learning miss opportunities to identify and address the kinds of struggles students face that lead to their underperformance, decision to drop out, or belief that they are not capable of succeeding. In a shared commitment to students’ equitable achievement of outcomes, every student counts—not solely those who persist and are the subjects of periodic assessment. Identifying and then addressing student learning challenges occur continuously to identify not only students’ initial learning challenges such as in writing or critical thinking, but also the challenges students face as courses and education experiences require them to build on, transfer, and integrate previous learning in diverse and more demanding contexts. Semester reporting of assessment results on dashboards (developments in dashboards are discussed in a chapter on technology discussed below) keeps all internal stakeholders on the frontline of students’ levels of achievement, course-by-course and education experience-by-education experience. Collaborative end-of-semester interrogation of these real-time results prompts the development of short- and longer-term interventions or practices to address enrolled students’ existing or persistent patterns of underperformance. Thus, all educators become engaged in improving student learning in a shared institutional commitment to equity.

At both the institution and program levels, the following six principles of real-time student assessment underlie a continuous commitment to monitor and address students’ academic challenges such as those related, for example, to attaining high achievement of the cognitive skills of synthesis and analysis or to applying previous learning into new contexts:                   

Six Principles of Real-time Student Assessment

Principle 1: Internally driven and motivated by institutional leaders and a commitment to students’ equitable progress towards achieving a high-quality degree;

Principle 2: Inclusive of internal stakeholders, including institutional leaders, administrators, full- and part-time faculty, and all other professionals in an institution’s network of experts who also contribute to and support students’ learning;

Principle 3: Bolstered by collaboration between faculty and an institution’s network of experts that harnesses others’ expertise and practices;

Principle 4: Anchored in continuous reporting and interrogation of assessment results—semester-by semester (sometimes by mid-semester) to monitor students’ equitable progress and address patterns of underperformance when they occur and as they persist in different contexts;

Principle 5: Responsive to students’ needs in the present tense in a commitment to all students’ progress toward attaining a high-quality degree;

Principle 6: Institutionally valued through a college’s or university’s rewards and recognition system

These principles do not dictate specific ways that institutions or programs integrate real-time student assessment into institutional cultures. In the book, five case studies illustrate various approaches institutions and programs have taken to integrate these six principles. However, closely monitoring students’ patterns of performance, interrogating patterns of underperformance, and taking actions to improve those patterns in the present tense are central processes in these case studies. All focus on supporting enrolled students’ equitable success. A chart following these cases argues for the 21st-century relevance of real-time assessment compared with more common assessment approaches that occur periodically. These more common approaches fall short in addressing the needs of all students, including those who struggle in between assessment time intervals.  Of importance as well is the fact that real-time assessment values what matters to faculty and other contributors to student learning:  their individual courses and education experiences. Reporting, interrogating, and nimbly using these assessment results occur at the end of each semester in the presence of other department- or program-level faculty, faculty who contribute to students’ general education outcomes, and representatives from the institution’s network of experts. This collaborative assessment approach deepens and broadens the all-important task of identifying immediate and longer-term strategies or practices to improve student learning. It marshals the experiences and practices of other faculty and experts. And, it provides colleagues with a real-time profile of students’ patterns of improved performance as well as patterns of underperformance, triggering the longitudinal need to address persistent challenges along students’ education pathway.

Real-time Student Assessment also identifies developments or refinements in technology that now contribute to the dexterity educators need to move with immediacy from evidence of student learning to actions to improve student learning. Developments to date

increase access to and interrogation of assessment data; provide real-time visual displays of student performance against agreed upon scoring rubrics or other metrics; identify invisible obstacles students face as they learn; or provide complementary data about student behaviors that enable an institution and its programs to change current practices and policies that inhibit students’ persistence to degree. Among the technologies discussed are the following:

  • Learning management systems (LMS) or other platforms that collect, synthesize, sort, and report real-time assessment results;
  • Assessment management systems (AMS) that also perform those tasks for an entire institution and its programs and display student performance against agreed upon scoring rubrics based on different filters, such as demographics or course level;
  • Adaptive learning platforms that individualize instruction by addressing learner challenges and report students’ course or module progress and final performance;
  • Web-based educational data-mining tools that provide new insights into institutional-, program-, and student-level variables that affect students’ progress or are predictive of their continued success; and
  • ePortfolios that provide a shared means for faculty, other contributors to student learning, and students themselves to gauge continuously students’ equitable progress toward attaining high-quality institution and program-level outcomes.

I fully recognize that real-time student assessment will unsettle some readers because it challenges an institution’s or its programs’ current assessment mind-set or established practices that satisfy external stakeholders’ needs. Reframing, even shifting, what already exists disrupts comfort levels. For that reason, the final chapter of this book identifies some initial approaches that institutions or programs might take—especially institutions with large enrollments. A narrowed, pilot-approach to real-time assessment can pave the way toward taking it to scale. Narrowed approaches might gauge students’ progress toward achieving quality-level outcomes in all high-risk courses or toward attaining increasingly higher performance levels in outcomes that remain difficult for students to attain, such as critical thinking. Case studies in the final chapter of the book illustrate how institutions and programs are developing or have developed real-time assessment processes. Several cases illustrate how real-time assessment effectively works in conjunction with other institutional efforts, such as early alert programs or effective advising practices that include recognizing a students’ immediate academic needs, referring that student to the appropriate campus resource, and monitoring that student’s subsequent improvement. Altogether these complementary efforts gauge students’ progress toward degree completion and prompt on-time interventions or practices to address enrolled students’ needs or obstacles.

For many institutions effectively using results—the action-oriented purpose of assessment—still remains their Achilles’ heel, as documented in NILOA’s 2014 report on institutional assessment practices (Kuh, Jankowski, Ikenberry, and Kinzie, p. 28). More effective use of assessment results can now occur when the unit of assessment analysis remains at the course and education experience levels.  Educators are not separated in time from their assessment results, from their students, or from their course design and pedagogy. Immediate synchronous interrogation of assessment results occurs within the various chronological contexts for student learning, leading, in turn, to specific context-based interventions or practices along students’ education pathways. 

Prioritizing real-time student assessment is now necessary to close persistent opportunity and graduation gaps between historically represented and underrepresented students; keep students on track to a degree; and assure that students achieve institution- and program-level outcomes at the high-quality levels necessary to transition into our 21st-century workforce and contribute to our democracy. Enrolled students are the immediate beneficiaries of this assessment process. What about future students? Surely, how institutions continuously improve enrolled students’ learning is relevant to the next class of students. 

                                     

References

Alverno College. (n.d.) Student Assessment-as-Learning. Retrieved from http://lampout1.alverno.edu/saal/

Association of American Colleges & Universities. (2014). VALUE: Valid assessment of learning in undergraduate education. Retrieved from www.aacu.org/value

Association of American Colleges & Universities. (2015). An Introduction to LEAP: Liberal education and America’s promise. Washington, DC: Author. Retrieved from https://www.aacu.org/sites/default/files/files/LEAP/IntroToLEAP2015.pdf

Federal Student Aid Office. (2013, March 19). Applying for Title IV eligibility for direct assessment (competency-based) programs. Retrieved from ifap.ed.gov/dpcletters/GEN1310.html

Kuh, G. D., Jankowski, N., Ikenberry, S. O., & Kinzie, J. (2014). Knowing what students know and can do: The current state of student learning outcomes assessment in US colleges and universities. Urbana, IL: University of Illinois and Indiana University, National Institute for Learning Outcomes Assessment.

Lumina Foundation. (2014). Degree Qualifications Profile. Retrieved from degreeprofile.org/read-the-dqp/the-degree qualifications%20profile/intellectual-skills

Lumina Foundation. (2009, April 8). Tuning USA: Lumina Foundation launches faculty-led process that will involve students and employers in linking college degrees to workplace relevance and students’ mastery of agreed-upon learning objectives. Retrieved from www.luminafoundation.org/news-and-events/tuning-usa-lumina-foundation-launches-faculty-led-process-that-will-involve-students-and-employers-in-linking-college-degrees-to-workplace-relevance-and-students-mastery-of-agreed-upon-learning-objectives

University Innovation Alliance. (n.d.). Vision and prospectus. Retrieved from www.theuia.org/sites/default/files/UIA-Vision-Prospectus.pdf

Real-time Student Assessment: Meeting the Imperative for Improved Time to Degree, Closing the Opportunity Gap, and Assuring Student Competencies for 21st-Century Needs will be published in February 2017 by Stylus Publishing, LLC.

.

Check out our past Viewpoints:

Real-time Student Assessment: Prioritizing Enrolled Students’ Equitable Progress toward Achieving a High-Quality Degree
Peggy Maki

Academic and Student Affairs Sides of the House: Can We Have an Open Concept Learning Design?
Darby Roberts

Just Assessment. Nothing More. Nothing Less.
Wayne Jacobson

Design for a Transparent and Engaging Assessment Website
Frederick Burrack and Chris Urban

Improvement Matters
Peter Felten

Working Together to Define and Measure Learning in the Disciplines
Amanda Cook, Richard Arum, and Josipa Roksa

The Simplicity of Cycles
Mary Catharine Lennon

Helping Faculty Use Assessment Data to Provide More Equitable Learning Experiences
Mary-Ann Winkelmes

Ignorance is Not Bliss: Implementation Fidelity and Learning Improvement
Sara J. Finney and Kristen L. Smith

Student Learning Outcomes Alignment through Academic and Student Affairs Partnerships
Susan Platt and Sharlene Sayegh

The Transformation of Higher Education in America: Understanding the Changing Landscape
Michael Bassis

Learning-Oriented Assessment in Practice
David Carless

Moving Beyond Anarchy to Build a New Field
Hamish Coats

The Tools of Intentional Colleges and Universities: The DQP, ELOs, and Tuning
Paul L. Gaston, Trustees Professor, Kent State University

Addressing Assessment Fatigue by Keeping the Focus on Learning
George Kuh and Pat Hutchings, NILOA

Evidence of Student Learning: What Counts and What Matters for Improvement
Pat Hutchings, Jillian Kinzie, and George D. Kuh, NILOA

Using Evidence to Make a Difference
Stan Ikenberry and George Kuh, NILOA

Assessment - More than Numbers
Sheri Barrett

Challenges and Opportunities in Assessing the Capstone Experience in Australia
Nicolette Lee

Making Assessment Count
Maggie Bailey

Some Thoughts on Assessing Intercultural Competence
Darla K. Deardorff

Catalyst for Learning: ePortfolio-Based Outcomes Assessment
Laura M. Gambino and Bret Eynon

The Interstate Passport: A New Framework for Transfer
Peter Quigley, Patricia Shea, and Robert Turner

College Ratings: What Lessons Can We Learn from Other Sectors?
Nicholas Hillman

Guidelines to Consider in Being Strategic about Assessment
Larry A. Braskamp and Mark E. Engberg

An "Uncommon" View of the Common Core
Paul L. Gaston

Involving Undergraduates in Assessment: Documenting Student Engagement in Flipped Classrooms
Adriana Signorini & Robert Oschner

The Surprisingly Useful Practice of Meta-Assessment
Keston H. Fulcher & Megan Rodgers Good

Student Invovlement in Assessment: A 3-Way Win
Josie Welsh

Internships: Fertile Ground for Cultivating Integrative Learning
Alan W. Grose

What if the VSA Morphed into the VST?
George Kuh

Where is Culture in Higher Education Assessment and Evaluation?
Nora Gannon-Slater, Stafford Hood, and Thomas Schwandt

Embedded Assessment and Evidence-Based Curriculum Mapping: The Promise of Learning Analytics
Jane M. Souza

The DQP and the Creation of the Transformative Education Program at St. Augustine University
St. Augustine University

Why Student Learning Outcomes Assessment is Key to the Future of MOOCs

Wallace Boston & Jennifer Stephens

Measuring Success in Internationalization: What are Students Learning?
Madeleine F. Green

Demonstrating How Career Services Contribute to Student Learning
Julia Panke Makela & Gail S. Rooney

The Culture Change Imperative for Learning Assessment
Richard H. Hersh & Richard P. Keeling

Comments on the Commentaries about "Seven Red Herrings"
Roger Benjamin

Ethics and Assessment: When the Test is Life Itself
Edward L. Queen

Discussing the Data, Making Meaning of the Results
Anne Goodsell Love

Faculty Concerns About Student Learning Outcomes Assessment
Janet Fontenot

What to Consider When Selecting an Assessment Management System
R. Stephen RiCharde

AAHE Principles of Good Practice: Aging Nicely A Letter from Pat Hutchings, Peter Ewell, and Trudy Banta

The State of Assessment of Learning Outcomes Eduardo M. Ochoa

What is Satisfactory Performance? Measuring Students and Measuring Programs with Rubrics
Patricia DeWitt

Being Confident about Results from Rubrics Thomas P. Judd, Charles Secolsky & Clayton Allen

What Assessment Personnel Need to Know About IRBs
Curtis R. Naser

How Assessment and Institutional Research Staff Can Help Faculty with Student Learning Outcomes Assessment
Laura Blasi

Why Assess Student Learning? What the Measuring Stick Series Revealed
Gloria F. Shenoy

Putting Myself to the Test
Ama Nyamekye

From Uniformity to Personalization: How to Get the Most Out of Assessment
Peter Stokes

Transparency Drives Learning at Rio Salado College
Vernon Smith

Navigating a Perfect Storm
Robert Connor

It is Time to Make our Academic Standards Clear
Paul E. Lingenfelter

In Search for Standard of Quality
Michael Bassis

Avoiding a Tragedy of the Commons in Postsecondary Education
Roger Benjamin