Important decisions are made on how people perform during assessments. In the mining industry staff members are only allowed on to the premise if they are compliant, that is if they have passed the required assessment. These assessments also have expiry dates and staff have to redo the course and assessments on a regular basis.
If an accident occurs investigators will refer to the assessments and ask which questions were answered correctly or incorrectly.
I have however rarely seen an active analysis of these questions by the business and the following questions are often not asked:
- Do the questions really measure what they were intended to measure?
- Are the questions reliable, in other words consistent in what they measure?
In a recent white paper by QuestionMark these concepts are discussed in detail.
This white paper draws upon the metaphor of a doctor using results from a blood tests to diagnose an illness and then prescribing a remedy. If the wrong test is ordered by the doctor delays will occur, or if the results are untrustworthy, serious consequences could result. Using this metaphor, it is easy to understand the personnel and organizational risks that can result from making decisions based on untrustworthy results. If an organization assesses someone’s knowledge, skill or competence for health and safety or regulatory compliance purposes, you need to ensure the assessment instrument is designed correctly and runs consistently.
The paper explains validity and reliability in detail. It also describes the process you should follow to improve your assessments which is plan, author, pilot, review, deliver and analyse.
This valuable report is available at: http://link.questionmark.com/l/26732/2014-12-09/2s9xhf
EPI-USE also offers courses in assessment and if you are interested in improving your assessments you are welcome to contact us.
Do you analyse your question results to improve your question banks?