The ratings you assign to items are used to calculate the Average Angoff of the items.
Once you are done rating the items on an exam form, you must compare your ratings with those of other SMEs.
- The Evaluate Your Ratings page allows you to view your own ratings as well as the ratings of other SMEs.
- A user rating is regarded as an "outlier" if it differs from the average rating by more than 15 and if it is greater than the standard deviation of the average rating ranges for all items in the exam form, and is displayed in red text.
- You are advised to change an outlier, since it affects the average range of the rating, subsequently giving rise to an unusually high or low Angoff value for an item.
- You can also decide whether to keep the rating assigned to items on the Rate Difficulty (Angoff) page or revise the ratings so they can be comparable with those of other SMEs.
- Once you have rated the items, you can click the link provided on the summary page to evaluate your ratings.
The Evaluate Your Ratings page is displayed as below.
The Evaluate Your Rating page includes the following features:
The rating you assigned the item on the Rate Difficulty (Angoff) page.
The range of the ratings for the item as assigned by all SMEs.
The average of the range of the ratings for the item as assigned by all SMEs.
A drop-down list that allows you to revise your ratings.
- If you need to revise your rating, select one of the ratings from the drop-down list under the Revise field.
Evaluate all the items. Once you are comfortable with the ratings of the items, click Finalize Ratings.