Navigation Menu

Click the "+" to see inside a chapter or use the search to the right.


Skip to end of metadata
Go to start of metadata

The ratings you assign to items are used to calculate the Average Angoff of the items.

Once you are done rating the items on an exam form, you must compare your ratings with those of other SMEs.

  • The Evaluate Your Ratings page allows you to view your own ratings as well as the ratings of other SMEs.
  • A user rating is regarded as an "outlier" if it differs from the average rating by more than 15 and if it is greater than the standard deviation of the average rating ranges for all items in the exam form, and is displayed in red text.
  • You are advised to change an outlier, since it affects the average range of the rating, subsequently giving rise to an unusually high or low Angoff value for an item.
  • You can also decide whether to keep the rating assigned to items on the Rate Difficulty (Angoff) page or revise the ratings so they can be comparable with those of other SMEs.
  • Once you have rated the items, you can click the link provided on the summary page to evaluate your ratings.

The Evaluate Your Ratings page is displayed as below.



The Evaluate Your Rating page includes the following features:




  1. The Stem of the rated item.
  2. Identifies if you answered the item correctly or not.
  3. The percentage of SMEs who have answered it correctly.

Your Rating

The rating you assigned the item on the Rate Difficulty (Angoff) page.


The range of the ratings for the item as assigned by all SMEs.

Avg. Range

The average of the range of the ratings for the item as assigned by all SMEs.


A drop-down list that allows you to revise your ratings.


  1. If you need to revise your rating, select one of the ratings from the drop-down list under the Revise field.
  2. Evaluate all the items. Once you are comfortable with the ratings of the items, click Finalize Ratings.


    If any of the ratings are below chance, the system displays a warning to confirm that the ratings should be saved.

    How can I track the progress of the users who are giving ratings?


    If you have been given Observer access, you can review the progress of users performing the ratings. For more information, see Exam Forms tab under Observe.


  • No labels