In this section, we will review: Tips on developing content for test items (below) and:
Here are the Top 10 recommendations to take into account when developing examination content:
1. Design Items to Reflect Specific Content/Skills
The classification system outlined in the exam blueprint specifies a number of content areas to be evaluated. Each item must be classified according to an appropriate area in the exam blueprint which specifies the content area that each item addresses. It should be recorded as specifically as possible. Each item must reflect a single specific knowledge or skill, as required in the exam specifications. Design items to include only important content, and do not include trivial content. The courts of the United States (and most western countries) require that all items be explicitly linked to a facet of professional performance.
2. Assess Necessary Items
Be meticulous in determining which items thoroughly evaluate an examinee’s understanding of the given content area. An effective method for generating items is to review all important topics and information related to the subject matter, focusing on sentences and statements that identify key ideas and necessary facts. By reviewing current research and other documents related to the exam subject, an item writer is less likely to overlook key topics.
3. Use Appropriate Vocabulary
Tailor the item’s vocabulary to the target audience. If candidates generally have high school educations, then the use of PhD-level vocabulary is inappropriate. The exam should evaluate a candidate’s knowledge of specific content areas, not his or her reading ability. Beware of overly difficult or potentially unfamiliar language. Finally, beware of overusing acronyms, and if there is any question about the meaning of an acronym be sure to define its meaning in parentheses.
4. Use Material in Novel Ways
According to Haladyna, Downing, and Rodriguez (2002), the novel use of familiar material can be an effective way to evaluate a candidate’s critical thinking and problem solving skills. They point out that to avoid testing only a candidate’s recall capacity, it is important to paraphrase course materials and lectures. Scenario-based items can provide a method for testing familiar material in fresh way.
5. Include References
When writing items, you must include a published reference where the content of each question can be verified. References should be comprised of standard, recognized texts used within the professional field from which the exam’s content is drawn. It can be helpful to examine these texts prior to developing items, recording the pages where useful question information is found. If possible, include more than one published reference for each item. Beware of relying on Internet sources as references, as source locations frequently change or vanish. Be sure to keep in mind that an exam containing even one unreferenced item is legally indefensible.
6. Avoid Ambiguous Content
Ambiguous, vague, and other non-specific content is ineffective in testing knowledge of specific content areas. Each item should have a specific purpose focused on evaluating a candidate’s knowledge of a specific content area. Questions which are too broad or too narrow in focus can be unclear for candidates.
7. Avoid Overlapping Item Content
Items shouldn’t make use of content used in other items on the exam. As each item should reflect a single specific knowledge or skill, ensuring that each item has separate content minimizes the risk of confusion.
8. Avoid Opinion-Based Content
While it is important to evaluate a candidate’s professional judgment, care should be taken to generate content following well-established guidelines. Items relying on the work of obscure and/or controversial professionals should be avoided, as should opinion-based or otherwise unsupported material.
9. Avoid Trick Questions
The intent of an exam is to assess a candidate’s knowledge and skill, not to determine how clever the candidate (or item writer) can be.
10. Avoid Insensitive Content
Effectively targeting an exam to its audience requires item writers to be sensitive to the ethnic, cultural, and professional groups that may make up a candidate pool. Sensitivity to cultural, religious, racial, and other groups is critical so as not to undermine an effective exam with insensitive content. Focus items on professionally related content. If an item contains content that would be unfamiliar or misleading, or if candidates would be more or less likely to get an item correct because of their group background, revise the language. Avoid cultural, religious, or similar cues that non-group members may miss. Do not include offensive or controversial content. Finally, ensure that no content gives a negative representation of cultural, religious, racial, or other groups.