Questionmark Corporation
10/21/2021 | News release | Distributed by Public on 10/21/2021 09:06
Mirror, mirror on the wall, who’s the fairest of them all?
Six approaches to making a workplace test or exam fair
Posted by John Kleeman, Founder
From an early age, we have an innate sense of unfairness. You can see this easily if you give two children pieces of chocolate and one piece is slightly bigger than the other!
Fairness itself is harder to pin down. The global assessment community is currently looking into the mirror to see if we can make testing fairer. I recently participated in a round table at the European ATP Conference with John Weiner of PSI and Andre Allen of Fifth Theory on Advancing Equity in Assessment which looked at fairness in testing. This article is inspired by this dialog, here are six approaches to make workplace quizzes, tests and exams fairer.
If you give tests, quizzes or exams as part of post-hire development, for compliance or job readiness purposes, or internal certification, how do you ensure they are free from bias? How confident are you to look in the mirror and determine that your testing program, if not the fairest of them all, is at least as fair as you can make it?
Here are six approaches to making tests and exams in the workplace fair.
1. Ensure all test takers are treated broadly equivalently
A principle of standardized testing is that all test takers should be treated equivalently ("equivalence of treatment"). The questions or tasks required to assess should be comparable, the administration environment should be comparable and the scoring and interpretation results should be comparable. The more you can make the testing process uniform, for example by using the same technology and the same procedures, the more likely it is that the results will be fair and trustworthy.
For example, if you are giving a compliance test to people in many different locations, the content of the test, the way that it is presented, the way it is scored and the impact of passing and failing should be broadly the same.
2. Ensure that passing the test requires only relevant skills
Every test has a purpose. For example, a health and safety test might measure whether someone can operate a machine safely, or an internal certification test might check if someone has the skills or competence to do an important job task. Ensure the test mirrors its purpose and does not include irrelevant items or aspects.
Psychometrically speaking, a test should not advantage or disadvantage individuals because of characteristics irrelevant to the construct being measured ("psychometric fairness").
For example, if typing quickly and reliably is not an important part of the skill being tested, don't require test takers to have to type lots of text to answer the test.
3. Follow legal guidelines
Particularly if test scores have consequences (e.g., hire, fire, promote), it's vital to follow legal rules on fairness ("legal fairness"). A key step in most jurisdictions is to construct a series of defensible steps between what is required to pass the test and to do a job role competently. You need to make a defensible argument that someone who does well on the test is more suitable for the job role than someone who does less well.
For example, a common technique is to conduct a job task analysis survey for a job role to determine what skills are needed in a job, and then use the survey results to build the test blueprint. This helps build the case that the test matches the job. See Questionmark's white paper, "Defensibility and Legal Certainty for Tests and Exams" for more on legal defensibility.
4. Felt fairness matters too
Fairness can be in the eye of the beholder. It is important that test takers and other stakeholders feel that the test is fair ("felt fairness"). Test takers who feel that a test is unfair may complain and may take measures to circumvent or cheat at the test. Even if the test seems to you objectively fair, if your employees and stakeholders think it's unfair, that's negative.
There can be many reasons for people to question fairness. One good practice is to ensure that people have good opportunities to learn the material needed to pass the test. And everyone can have a bad day - consider how you permit retakes. And listen to input or complaints from test takers. Considering felt fairness will not only make the test fairer but may also reduce test fraud. If test takers believe the test is fair, they are less likely to rationalize attempts to cheat it.
5. Be inclusive
Sometimes we can be inadvertently biased against some people in the workplace without meaning to.
Bias can be a failure to provide an accessible solution to people who have one of many different disabilities (visual, cognitive, auditory, physical and more). Use the capabilities of technology to aid people with disability.
Bias can be a failure to take account that someone's native language is not the same as that of the test. Consider offering extra time to such people, or in some cases translating the test.
Bias can be using test items which are culturally orientated to one demographic, e.g. using words or concepts that don't make sense in other cultures. For example, in the Northern hemisphere, we understand the concept of "mowing the lawn" but that is a foreign concept in the Middle East. Ensure that your item writer training takes this into account and that the questions are reviewed by a culturally diverse set of reviewers.
In general, consider the audience of test takers in your organization and ensure that you are being inclusive.
6. Don't be unfair
Lastly, avoid unfairness.
People are not machines. Keep an open mind if test takers raise issues of unfairness, and reasonably consider their requests. Formal programs should have an appeal procedure that is open to test takers who want to raise issues.
Remember also that some people have difficulty with formal tests and exams, either because of test anxiety, bad experiences in the past or inexperience. Be open to such issues. Is there an alternative way that someone can demonstrate their skill or knowledge? Or is there some help or accommodation you can provide to assist someone who has challenges.
Be open and flexible to possible unfairness that arises as you deliver and improve your program.
There probably cannot be a test that is the fairest of them all, nor even a test that is 100% fair. After all, magic mirrors do not exist in the real world. But if you follow these six principles, and seek to continually improve, you are likely to have a test which is fair enough:
- Equivalence of treatment
- Psychometric fairness
- Legal fairness
- Felt fairness
- Being inclusive
- Not being unfair
For more information or a demo of the Questionmark assessment management system to help you deliver fair tests to your workforce, visit https://www.questionmark.com/demo
John is the Founder of Questionmark. He wrote the first version of the Questionmark assessment software system and then founded Questionmark in 1988 to market, develop and support it. John has been heavily involved in assessment software development for over 30 years and has also participated in several standards initiatives.