The selection of participants for a standard-setting study is important because the committee not only affects the resulting cut score, but also the credibility of the standard.
Individuals who establish cut scores MUST be qualified to make the judgments required of them. A single panel of participants should be sufficiently representative and large to assure that a similar cut score would be obtained if the study were replicated. One should pre-define the appropriate qualifications of participants and stick to your standard.
It is important to be certain that the makeup of your committee is an accurate representation of the testing population. An appropriate cross section of the field regarding, age, gender, geography and tenure are all important. These demographics were likely determined at the job/task analysis phase of your program’s development.
There is no magic number of participants that should make up the standard setting committee. However, the larger the testing population and stakeholder groups, the more participants should be selected. The fewer the participants, the less likely that the process can be replicated. It is recommended that greater than five participants be used for a smaller program and that no more than 30 is necessary for a larger program.
When it comes to participant selection, one method of simplifying the process is to appoint your most proficient participants as “Nominators.” Ask that these individuals recruit potential participants to serve on the panel that they consider to meet the eligibility that was determined.
Train your participants!
It is important to document all standard setting procedures that are used to train participants. Studies have found inconsistency in results of using groups where training was insufficient. It is unlikely that participants will understand or correctly perform the required tasks without the benefit of proper training.
In the training documentation, it is important to delineate the tasks required of the trainee, clearly define the knowledge, skills and abilities underlying performance of the tasks, and most importantly, instruct participants on how to acquire the defined knowledge, skills and abilities.
In order to perform their role effectively, participants must have an understanding of the context of the standard setting. It requires familiarity with:
- The purpose of the examination
- How the examination was developed
- The content to be covered
- The reason a standard is being set
- The consequences of the standard
The more context of the assessment that you provide, the more they are likely to appreciate the benefits and potential limitations of the assessment enterprise.