Navigation Menu

Click the "+" to see inside a chapter or use the search to the right.

.

Skip to end of metadata
Go to start of metadata

The Standard Setting component is ExamDeveloper's “best practice” method of setting a passing standard for an examination. The standard setting process is performed by a group of Subject Matter Experts (SME) and is best performed while everyone is on the system at the same time.

When an Organization sets a passing standard for an exam form, there are several factors they consider, including:

  • What is the difficulty of the exam form? SMEs developing a high-stakes exam usually write items that span a wide range of difficulty, so it is advisable to document a quantitative evaluation of the difficulty of the exam form.
  • What is the minimum level of performance that can be considered "competent"?
  • What is the experience level of the target population with the material being tested?
  • The process is performed using a representative committee of SMEs, typically ranging between eight and fifteen members.

What does the term 'Minimum Competence' mean?

Icon

A concept that describes what the minimally acceptable individual should be able to do, or should know, when performing the job. This individual is typically contrasted with an Excellent or Superior individual. A minimally competent worker is defined according to observable work behaviors that demonstrates they possess just barely enough knowledge, skills, or abilities to be called "competent".

 

Certification exams assess the required knowledge and skills of candidates in regards to professional performance, so it is critical that the passing standard be determined through considered analysis.

  • There are established methods for determining passing standards: the Angoff Method, the Bookmark Method, Ebel's Method, and more.
  • A methodical approach to the process ensures a reproducible, quantifiable examination passing point.
  • ExamDeveloper uses the Angoff Method to establish a defensible passing standard.

 

Icon

What is the Angoff Method?
The Angoff method is one of the most widely used systems for standard setting. First, a panel of SMEs reach a determination about the expected knowledge and abilities of the minimally competent examinee. Then they consider each item on the exam form and assess the probability such an examinee would answer each item correctly. These probability scores are then averaged for each item, and the average of the the total probability scores is used to determine the passing score for the exam form.

 

ExamDeveloper allows you to set the passing standard for an exam form in the following three steps:

  • Take the Exam: The SMEs take the examination exactly as a candidate would do.
  • Rate the Difficulty: For each item, the SME sees the correct answer, what they chose as a response, and the percent of other SMEs who answered the item correctly. They then estimate the number of minimally competent people who would be able to answer it correctly.
  • Review Ratings: The SME then evaluates his/her ratings in comparison to other SME ratings for each item. If the SME feels his/her rating for an item is overly harsh or lenient, he/she have the opportunity to adjust the ratings.

Do not change items while standard setting is in progress

Icon

It is important that you do not change items in any way (item text, Blueprint ID, Metadata, etc.) while a standing setting study is in progress, as some participants' ratings are stored against the old version of the item and others are stored with the new version. When this occurs, participants may experience errors when submitting their ratings (requiring them to re-log in to ExamDeveloper) and reports may not reflect all of the recorded data.

Icon

Do not use your actual exam form to test workflow or to look at screens; create a sample exam form. Once ratings are given to items, they cannot be removed and they show on reports.

When using "Set Standard", turn off Angoff collection on Validate page.

Icon

For projects that use both the "Validate" and "Set Standard" components, it is recommended that you turn off the Collect Angoff data during validation project setting. Collecting the data in both areas is redundant, and the Angoff information collected during the Validation stage is overwritten with the Set Standard data.

Set standards for an exam form

Select the Create Exam Form → Set Standard menu. The Summary tab opens on the Exam Listing page.

  

 

The Exams Listing page includes the following features:

Option

Description

Exam Form Name

The Name of the exam form.

Created By

Name of the person who created the exam form.

Status

Indicates whether the exam is being drafted or has been published.

Set Standard

Click Set Standard to start the process of setting standards.

Take ExamClick Take Exam to start the Take Exam process.
ReportRun a report with the ratings of the users.

Functionality of the Report button

Clicking the Report button downloads an Excel file with Set Standard data for exams, which have been rated by SME(s) via the Set Standard process.

The report consists of a set of similar columns for each SME:

  • Question Number: This is the Question ID of items in the exam form.
  • Response of SMEx: This is the Answer Option chosen for the item by the SME.
  • Final Difficulty Rating by SMEx: This is the Difficulty Rating assigned to an item by the SME.
  • Comment of SMEx: This is the comment (if any) added by the SME to an item.

    Icon
    In each of the above cases, the “x” suffix of SME is an integer starting from 1, which is incremented as additional SMEs apply the Set Standard process to the Exam Form.

     

 

 

  • No labels