- Introduction to Standard Setting
- How to Set up a Standard Setting Meeting
- Different Standard Setting Techniques: Absolute and Compromise Methods
- Associated Legal Issues
Here we revisit our cyclical model of examination exam development. Once an examination exam has been assembled, we need to set a standard to determine the appropriate level of difficulty.
Recall your days in school when teachers set the universal "70%" as the minimum score needed in order to pass an exam. Although this percentage may be sufficient for a grade school science class, using this or any passing score without sound, research-backed methodology is the easiest way to invite legal challenges to your exam program. For all intensive intents and purposes, 70% is just an arbitrary number. It is unlikely that your teachers were psychometrically determining that each test administered was equal in difficulty.
The purpose of a standard setting is to determine the exact point that separates those candidates who know the material and those who do not. We want to be confident that good performance on an examination will correlate exam correlates to high competence on a job, for example. This is a difficult process and is open to much legal scrutiny. In fact, it is one of the most common legal challenges for employment exams.
Image RemovedImage Added
Norm-Referenced vs. Criterion-Referenced
Tests Exams can be categorized into two major groups: norm-referenced tests exams and criterion-referenced testsexams. These two types differ in their intended purposes, the way in which content is selected, and the scoring process which defines how the test results must be interpreted.
A norm-referenced test exam is a type of testexam, assessment, or evaluation which yields an estimate of the position of the tested individual in a predefined population, with respect to the trait being measured. This estimate is derived from the analysis of test exam scores and possibly other relevant data from a sample drawn from the population. The term normative assessment refers to the process of comparing one test-taker to his or her peers. Norm referenced examinations exams are designed to highlight achievement differences between and among students to produce a dependable rank order across a continuum of achievement from high achievers to low achievers. Schools might want to classify students in this way so that they can be properly placed in remedial or gifted programs. For this purpose, they are fine, but they are not appropriate for employment examinationsexams.
On the contrary, a criterion-referenced test exam is one that provides for translating test scores into a statement about the behavior to be expected of a person with that score or their relationship to a specified subject matter. The objective is simply to see whether the student has learned the material. Criterion-referenced exams report how well candidates are doing relative to a pre-determined performance level on a specified set of educational goals or outcomes included in the school, district, or state curriculum.
Norm Referenced vs. Criterion Referenced
Norm-referenced methods are based on a comparison among the performances of examineescandidates. Using this method, a set proportion of candidates fails regardless of how well they perform. For example, the top 70% pass. Here is a sample test score distribution that applies the norm referenced method. The red shaded portion that illustrates the lowest 30% of candidates are the pre-determined cutoff point.
Image RemovedImage Added
Criterion Referenced exams are used to determine whether candidates know the subject matter or they do not. Here is a test score distribution using the criterion-referenced method. You can see the exact intersection of the group that performed well and the group that performed poorly. This is designed to be the point of minimum competence.
Image RemovedImage Added
When organizations set a passing standard for their exam, there are several important factors they consider:
Subject matter experts developing a high-stakes exam usually write questions items with a wide range of difficulty, from very easy to very difficult. As such, it is advisable to document a quantitative evaluation of the exam difficulty, even if it is based on opinions from subject matter experts.
Observations of employees on the job will typically yield a range of performances, from exceptional (the superstars) to inadequate (the incompetent). The goal is to establish the minimum set of knowledge and skills someone would need needs to possess in order to be labeled "competent" in regards to the job.
A passing standard set too high will produce produces too few practitioners to adequately meet the needs of the organization, industry, or public. If too few people are meeting the performance standard, it may be advisable to revisit its appropriateness in regards to the exam’s objectives.