Pre/Post Assesment: Aligning Assessment Strategies for Multiple Student Populations
Created for The Center for International Education at Juniata College, Huntington, PA
August, 2013
Objectives
The objectives of this project were to map existing learning objectives through completion of Juniata College’s Desired Intercultural Competencies rubric and to align assessment of multiple student populations through the creation of core assessment questions that correspond to the rubric.
Rubric
Juniata’s Desired Intercultural Competencies Rubric was developed by Juniata faculty who created the desired competencies, sorted into six groups. These six groups examine a student’s ability to Remember, Understand, Apply, Analyze, Evaluate and Create in an intercultural context. Completing the rubric entailed establishing how a student could demonstrate that he or she had met the four levels of accomplishment, which are Extensive, Moderate, Minimal, and Inadequate.
As recommended by Lisa Wyatt Knowlton and Cynthia C. Phillips in their work, The Logic Model Guidebook, on modeling learning outcomes, I used the SMART concept (Specific, Measurable, Achievable, Relevant, and Time-bound) to clarify what it might look like for a student to demonstrate achievement at each level. In the example below, it is very easy to see that a student at an extensive level of achievement can correctly label at least thirty different countries spanning five continents on a world map, while a student that has demonstrated a minimal level of achievement can label between ten and twenty different countries spanning three different continents on a map.
In some learning outcomes, applying the SMART concept is a little more difficult. In the example below, students who demonstrate an extensive understanding of their own culture can provide a profound understanding of major aspects and smaller nuances of their culture. As recommended by Darla Deardorff in her work, Implementing Intercultural Competence Assessment, it is helpful to establish definitions of key words (such as “profound,” “major aspects,” and “smaller nuances”) that are agreed upon by each person who is administering and evaluating assessment results. Doing so will ensure that all students are assessed using the same definitions and helps to keep assessment results valid and reliable.
Core Questions
The purpose of creating core questions to assess the desired intercultural competencies is so that answers to these questions can be compared across multiple student populations. (This includes incoming first year students, graduating students, returned study abroad students, international students, and Global Village Residents.)
As shown in the examples below, these questions were formatted to be self-reporting (asking students what they think they know or what they think they can do) by asking to what extent they agree or disagree with each statement by circling a number one through eight. Many questions also directly ask students to demonstrate their knowledge or skill by answering a corresponding short-answer question. As Deardorff also recommends, integrating both direct and self-reported information will help to create a holistic picture of each student’s level of achievement.
According to Barbara Moskal and John Leydens in their work “Scoring Rubric Development: Validity and Reliability,” a helpful way to test the validity and reliability of the information yielded by these questions and rubric is to compare their answers and subsequent rubric placement to the answers and rubric placement yielded by different methods of assessment that evaluate the same content. This is also called parallel or equivalence reliability. For example, a student who marks an eight for the statement “I believe that the best way to understand another person’s feelings is to experience those feelings myself,” would also exhibit a high level of empathy when interactions with others are observed or when participating in class discussion that addresses intercultural topics. Over time, formatting and wording of questions should be revisited and revised to maintain or improve validity and reliability.
Recommendations
Some studies that compared test results from dynamic and static testing in children, suggest that students who are made aware of desired learning outcomes and their progress toward achieving those outcomes tend to perform at a higher level than those students who are unaware of desired outcomes. Therefore, I suggest that students not only be made aware of Juniata’s desired intercultural competencies, but are also given the opportunity to become familiar with the four levels of achievement. Doing so will give students a better understanding of what is expected of them, as well as help each student to better understand his or her own progress toward achievement.
The best way to utilize the information yielded by these questions is to integrate the use of pre and post tests as one component of a larger assessment plan. Use of summative tools like pre and post tests provide a snapshot in time of a students development within the learning outcomes context. But this does not provide the fuller picture of the evolution of the student’s development. As reflected in Deardorff’s Implementing Intercultural Competence Assessment, emerging assessment practices in international education call for the use of multiple, complimentary assessment methods. Using formative tools, such as blogs, e-portfolios, or course work, in conjunction with pre and post tests will provide a much more holistic picture of student development and achievement of Juniata’s desired intercultural competencies.
