Click the "+" to see inside a chapter or use the search to the right.

.

# 4.3.2 Rate the Difficulty Level of Items

ExamDeveloper uses the Angoff method for setting standards. In this method, each Subject Matter Expert (SME) rates each item by selecting the proportion of minimally competent people who will correctly answer the item. The ratings of all the SMEs are taken into account in assigning a difficulty level to the item.

Icon

One of the simplest statistical ways of measuring difficulty is the percentage of people who answered the item correctly in previous exam administrations. For organizations that build multiple versions of an examination, it is advisable to select items such that the difficulty of one exam form version is equal to that of the others.

## Rate the difficulty of items you’ve previously answered

1. After Taking the Test and answering all the items, you can rate the difficulty of the items by clicking the click here link provided on the Exam Result page.

Icon

The score is calculated as: Score = sum of weights of all correctly selected Options of all the items on the exam form.

If weight is not given for a correct Option, then for items of types MCQ Multiple Correct Options and Multiple True/False, the SME is given the score 1 for each correctly selected Option and 0 for incorrectly selected Option(s).

2. The Rate Difficulty (Angoff) page is displayed.

The Rate Difficulty (Angoff) page displays the following details for each item of the exam form:

Field

Description

Question # of #

The sequence number of the item as displayed in the exam form and the total number of items in the exam form

The correct Answer Option for the item

You Chose

The Answer Option that you chose for the item

The number of SMEs who answered the item correctly and the corresponding percentage

Icon

This is a project setting the Project Manager can disable, so this may not be seen in all projects.

Question

The item with its Answer Options

3. To rate the difficulty of the Question: From the Percent of MINIMALLY COMPETENT people who would answer correctly drop-down list, select out of 100, the number of minimally competent people that are likely to answer correctly. If this number is high, the difficulty level of the item is low, and vice-versa.

What is the range for Item Difficulty?

Icon

The range of difficulty ratings is shown as 1-100. The system calculates the chance for random guessing. E.g., if the item has 4 Answer Options, the rating would generally not be below 25 since a candidate can randomly guess and get the item correct 25% of the time. If a rating below that chance is given, the system displays a warning message stating that the rating is below chance. The rating can still be saved by clicking the OK button on the pop-up message.

4. Click Next for the next item. Your rating is saved upon clicking Next
5. You can also navigate through the item stack using:

Option

Description

Previous

Click Previous to view the previous item

Next

Click Next to view the next item

Go to

Select the item you want to view from the list

Icon

Unrated items are displayed in red and have an asterisk next to them. All items that have been answered are displayed in black.

6. Repeat steps 1 through 3 to answer all of the items.
7. Click Finished to complete the rating process. The Rate Difficulty (Angoff) page opens.

Icon

If you have not answered all the items from the exam, the Rate Difficulty (Angoff) page states this and lists the unanswered items with their Question IDs and Question Stem text. You must click the clicking here link to go back and answer the items before you can proceed any further.

8. Once you are done rating all the items, click the click here link provided on the Rate Difficulty (Angoff) page to evaluate your ratings.

Icon

If an item is part of a case, the Exhibit text is listed above the item.

• No labels