Navigation Menu

Click the "+" to see inside a chapter or use the search to the right.

.

Skip to end of metadata
Go to start of metadata

After the domains are weighted and averaged by the SME panel, what's next? Validation!

This must be done to ensure that the domains, task statements, and knowledge and skill statements are acceptable by others in the profession. In order to confirm (or modify) a panel’s work, a validation study is the next necessary step and is highly recommended. You certainly hope that the work performed by your initial panel was perfect, but we all understand that's rarely the case. The validation study serves as the check and balance element of the JTA by the population at large.

How does one validate the content outline?

Icon
  • Second Panel
  • Focus Groups
  • Literature Review
  • Survey

A variety of procedures can be employed to execute this process:

For example, one can assemble a second panel to validate the work performed by the original panel, conduct a series of focus groups, or conduct a comprehensive literature review. Perhaps the most common technique is to format the meeting’s output into a survey. Such a survey is generally distributed to thousands of stakeholders in the appropriate field.

These stakeholders then have an opportunity to validate each element of the content that has been proposed using a variety of validation scales.

Who validates the JTA?

Icon

The JTA is sent out to other professionals in the field who them will review it and make sure that all parts of the job are covered and addressed. One must strive for a survey population that is representative of the industry population. A good response rate is around 25% of total recipients. You may want to get creative with encouraging survey completions. If folks get a "free gift", continuing education credits or a discount on the fees for the next industry conference, they will be more likely to complete the survey. The more data you collect, the more solid the foundation of your examination.

Icon

Part of the JTA survey is a demographic section to allow for analysis of potential sub-group differences in the target population to see if there are differences in how people perceive the work. One variable which potentially could impact how a person views related duties and requirements is his or her amount of job experience. As people gain more job experience, their job roles can change to include more complex or higher-level tasks. Other common variables include education, salary, size of organization, and number of employees supervised. As such, it is important to compare the task ratings of individuals across each of the variables to make sure that everyone views the importance, criticality, and frequency similarly.

 

The field professionals will rate the domains and tasks that the SMEs created on the same requirements, Importance, Criticality, and Frequency.

After the survey results are returned back, the average of all the ratings, both the SMEs and the field professionals are compiled and averaged out. With the average, the blueprint for the exam will be created. These averages show how many questions should be dedicated to each domain on the exam. The higher the rating, the more questions that should cover that topic.

After the blueprint is completed, the writing part of the exam can begin.

 

Icon

Validation Survey: Summary

  • Convert the JTA outline to a survey
  • Administer to sample of practitioners in the field -  Must be representative of the field (demographics, experience, education, etc.)
  • Ask respondents to rate the importance, criticality and frequency of each statement in the JTA outline
  • Combine ratings into a “Relevance Metric”
  • Compare responses with that of the original panel

 

Here is a snapshot of typical validation scales and other examples that one can use in a validation survey:

TYPICAL VALIDATION SCALES

Typical Validation Scales

Criteria

 Rating

Importance:

How essential is this activity during competent performance of this profession?

0        1        2        3        4

(not important to most important)

Criticality:

How much harm can be caused by an inability to perform this activity during competent performance of this profession? (e.g. legal, financial, psychological, physical, or other forms of harm)

0        1        2        3        4
(no harm to extreme harm)

Frequency:

How frequently does a competent professional spend time executing this activity during competent performance of this profession?

0        1        2        3        4
(never performed to very frequent)

 

 

Icon

These validation scales can also be appropriate when validating training content.

 

 

OTHER VALIDATION SCALES

 

Other Validation Scales

Criteria

 Rating

Need for Instruction:

How necessary is it that practitioners receive training about this duty or procedure?

0        1        2        3        4

(not necessary to essential)

Difficulty of Learning:

How difficult is it for practitioners to gain competence with this duty or procedure? 

0        1        2        3        4
(not difficult to very difficult)

Point of Acquisition:

At what point in their careers do practitioners acquire this duty or procedure? 

0        1        2        3        4
(before 1st day on job to late in career)

 

 

  • No labels