Design Training for ReviewersDescription
The training of graduation portfolio reviewers is essential to establishing a reliable scoring process. A graduation portfolio is considered to be reliably or consistently scored if it is awarded the same score regardless of which student completes it, who reviews it, or when it is judged.
Things to Consider
The goal of graduation portfolio reviewer training is to ensure that scoring criteria are applied uniformly to each graduation portfolio and that subjective application of scoring criteria is minimized. Activities in a graduation portfolio reviewer training session should focus on:
- Increasing scorers' familiarity with the GSEs and relevant standards
- Learning to locate evidence in a portfolio
- Applying the rubric and making a judgment based on standards
- Recognizing personal biases
- Experiencing a range of performances in portfolios
- Providing scorers with ample practice and feedback
- Achieving demonstrated agreement in scoring
- Developing speed and confidence in graduation portfolio scoring
(Adapted from Collins, A. (2002). Lessons Learned, Five Years of Science at INTASC. Paper presented at the 2002 Annual International Conference of the Association for the Education of Teachers in Science, Charlotte, NC. and Select or Design Assessments That Elicit Established Outcomes. North Central Regional Educational Laboratory. Available: 
http://www.ncrel.org/sdrs/areas/issues/methods/assment/as7sele2.htm)
Tools
School Roles and Responsibilities
This tool describes the preparation and materials required for conducting a graduation portfolio judge training session. It is important to provide the tools (rubrics etc.) and sample graduation portfolios for reviewers to practice on. One important aspect of training is to familiarize the reviewers with the requirements of the portfolio and the types of tasks that will comprise the portfolio. This process can be used as is or used as a model.
Judge Roles and Responsibilities
This tool outlines the roles and responsibilities of reviewers of portfolios. It comes from the National Center for Research, Evaluation, Standards and Student Testing (CRESST). For current information and publications from CRESST, visit http://cresst96.cse.ucla.edu/.
Graduation Portfolio Reviewer Training Design
This is a sample training design for reviewers that you can modify to meet the needs of your school. This sample was provided by the National Center for Research, Evaluation, Standards and Student Testing (CRESST). For current information and publications from CRESST, visit http://cresst96.cse.ucla.edu/.
Portfolio Score Sheet
This is a sample score sheet for reviewers of portfolios that you can modify to meet the needs of your system. This sample comes from the National Center for Research, Evaluation, Standards and Student Testing (CRESST). For current information and publications from CRESST, visit http://cresst96.cse.ucla.edu/.
This tool can be used during graduation portfolio judge training sessions and provides a place for trainees to record the scores they assign and notes they take as they review sample graduation portfolios.
Rater Agreement Summary Sheet
This sample rater agreement summary is a tool used to calculate the degree of rater agreement. The specific information provided in this form is only an example, and the form should be modified to meet the needs of your school.
This sample rater agreement form is an adapted version from the National Center for Research, Evaluation, Standards and Student Testing (CRESST). For current information and publications from CRESST, visit http://cresst96.cse.ucla.edu/.
Remember that in order to make the review of the portfolios fair and reliable the reviewers need to use the rubrics and process and come to similar judgments as to the sufficiency or quality of the portfolio contents.
