The previous section provided detailed best practices on developing exam content. This section addresses how one should enforce frequent reviews of items and what to look for.
A key step in the development process is to have all items reviewed by several subject matter experts for completeness, accuracy, and relevance. One big misconception of high-stakes exams is that they are written by one or a small number of people working independently.
Exam development is a group activity which is improved when many many people participate in the process. If any reviewer has an issue with the wording, they need to have the opportunity to revise the item and offer it to others until everyone approves of the item. Typically, at least three people besides the original author, must approve the item wording before it is eligible to be placed on an exam.
The review process for content development is very important and can appear exhaustive. Depending on the resources one may execute a bias review, a content review, a psychometric review and an editorial review.
- A content review is likely performed by a subject matter expert who can determine whether industry-specific language or references appear correctly. Below, you will see a helpful checklist of applicable items to confirm.
- A sensitivity review takes elements of the target audience into account like reading level. This can be achieved using the second checklist below.
- A psychometric review is performed by a test development professional who focuses on the quality of the writing, not the content, that is, whether the item conforms to psychometric best practices.
- Finally, an editorial review is achieved by an editor who has expertise in grammar, spelling and sentence structure. Occasionally, this review can be combined with the psychometric, content or sensitivity review.
Were best practices in stem development followed?
Is content of the appropriate difficulty for the target audience?
Is the language used clear, concise and appropriate?
Is the content appropriately linked to the curriculum or classification system?
Is the marked key the only correct answer?
Are the distractors plausible, but still incorrect?
Is there a verified reference?
Is the knowledge tested by the item important to illustrate competent practice?
Does the item use vocabulary that has multiple definitions?
Does the item examine controversial subject matter?
Does the item address subject matter that each candidate would have had access to in their education?
Does the item address content that may be foreign to some groups?
Is the language used overly difficult?
Could the language be regarded as demeaning or offensive?
Does the content reinforce stereotypical extensions of a group?
Is there perceived bias based on race, ethnicity, gender or socioeconomic status?