|
| |
 |
|
| VIEWPOINT |
|
| Year : 2018 | Volume
: 1
| Issue : 1 | Page : 4-6 |
|
Technology, innovation, and progress in testing: Eight barriers to change
David Foster
Chair and CEO of Caveon Test Security, United States
| Date of Web Publication | 1-Oct-2018 |
Correspondence Address:
Dr. David Foster
Caveon Test Security
United States
 Source of Support: None, Conflict of Interest: None
DOI: 10.4103/EHP.EHP_9_18
This “Perspectives” article involves the reflections of Dr. David Foster based on his nearly 40-year working experience in the assessment arena. Foster argues that, despite many innovative technology solutions, the assessment field has failed to introduce technology in creative, effective ways. This is evidenced by the fact that most modern assessments used in the high-stakes testing industry involve paper-and-pencil tests administered on computers. Foster identifies eight common reasons (and excuses) used to reject change and accept the status quo, thereby forcing the assessment field to live in the past.
Keywords: Assessment, certification, change, licensure, medical education, psychometrics
How to cite this article:
Foster D. Technology, innovation, and progress in testing: Eight barriers to change. Educ Health Prof 2018;1:4-6 |
A famous cartoon once captured the reality of change. In the comic, a speaker stands behind a podium and asks, “who wants change?” Virtually every hand in the audience went up. The speaker then asks, “who wants to change?” Sadly, no hands went up. Medical and health professions education, everything from medical training programs to medical licensure and certification, presents similar challenges. Most individuals seem unhappy with their assessment practices and want change, but relatively few seem willing to change.
When most of us think of change, we immediately think technology. The role of technology is undoubtedly one of the greatest change agents, as it often forces change upon even the most reluctant. The problem with technology in the assessment arena isn't so much the lack of technology, but rather our failure to introduce technology in a creative, effective way. Based on a personal review of high-stakes exams built today in the licensure and certification fields, I observed that approximately 99% are simply paper-and-pencil exams that are administered on computers. Although many technology solutions exist, we continue to live in the past and it is not doing us any favors.
Having worked in the broader assessment arena for nearly 40 years, I have often wondered why change is so slow. For example, why don't more people embrace creative approaches to testing? Why don't we have more adaptive tests, an innovative method that has been around for over 30 years? Why are the vast majority of items used in tests still traditional multiple-choice questions (MCQs) with four options and one correct answer? Why aren't scores provided immediately for most tests? Why aren't essays scored automatically and quickly instead of using raters? Why aren't more tests given on demand instead of on a few days a year? Why do we travel to a location to take a test rather than take it in our own home or office? Why does it take weeks or months to fix a compromised or broken exam? Why are we still suffering a growing number of security breaches? Below, I am proposing a list of “Eight Barriers to Change.” While this list certainly is not exhaustive, it does capture the most common barriers that I have observed over the course of my career. It is my hope that readers will recognize one or more of these barriers and realize that the barriers really are not that large at all.
| Eight Barriers to Change | |  |
#1: “We need to stick with the standards.” Some people confuse traditional long-standing methods and processes with standards. They would argue that, as a testing standard, traditional MCQs cannot and should not ever change. Of course, multiple-choice testing is simply a method, not a standard. If you are unsure, a standard is a goal or target to achieve, not a way to get there.
#2: “I'm good at what I do.” People are comfortable doing what they have learned to do and what they have always been doing. Change is always difficult.
#3: “How do I know it will work?” Some people are afraid of what will happen if they try new things. They are unsure whether a new approach will bring the promised benefits, and opt to play it safe and avoid new risks.
#4: “It needs more research.” Some people rely on the solid, but fairly slow and intermittent process of scientific research.
#5: “Who else is doing it?” Some people are late adopters, willing to let others blaze the trail. They do not want to try new methods without being confident that it has already worked for others.
#6: “I don't have the time.” A person may not have the time to learn about and understand the value of new tools and procedures.
#7: “That's hard to do.” Using and implementing new technology can take substantial time and resources, particularly at the beginning. Some individuals may not be willing to undertake the initial work it requires to implement new ideas.
#8: “It scares me a bit.” It's true that developing and using a new process or tool may not immediately result in a positive experience. That's the risk one takes with innovation. For some, it is better to avoid that risk altogether.
| Examples of Change | | |
I have used the title of psychometrician often during my career, but it never seemed to fit me well. I think a better title would be psychometric technologist. The title describes a person who believes that technology can make testing easier, less expensive, more helpful to the society, less risky, and more fun. Regarding the value of innovating, all I can really offer is my testimonial–based on my personal experiences–that using technology, even technology that has never been tried before, can have amazing positive effects. During my career, I've had the opportunity to encounter some significant testing challenges, roadblocks, and problems. There must be 20–30 clear examples that immediately come to mind. Let me describe a few of them.
In about 1993, when I was creating certification exams for Novell, a prominent software company at the time, we were getting hammered by our candidates and partners who complained that our exams were trivial, measuring textbook learning instead of real-world experience. Of course, the criticisms were mostly valid. It was clear that, instead of asking about how to do something using Novell software, it would be better to ask the candidate to actually complete tasks in the software. I received the go-ahead to create a simulation of NetWare, Novell's flagship networking software. The simulation interacted with the exam administration system and our exam development system so that it was easy to create items that asked the candidate to complete tasks. Scoring was straightforward, and the exam took no longer than the earlier version. We still used MCQs where appropriate to measure certain skills. To make a long story short, after implementing the new exam, the complaints about Novell's exams virtually disappeared overnight. The simulation cost about $250,000 to build, but was worth every penny. It was used in many certification exams, practice exams, and quizzes following training courses. Later, psychometric analyses revealed that the items performed better than we hoped. Further, our candidates and partners were thrilled with the result. Follow-up validity studies confirmed that certified candidates were well prepared to work for Novell clients and partners.
The use of innovative technology was motivated by a problem, and at the time, there was no research, no experience, and no tools available to help. My only choice to solve this dilemma was to take the risk and move forward.
Here is an example solution for a simpler problem. At around the same time, Novell candidates experienced some difficulty with the translation of specific words in exams. It seems some technical words that are usually well known in English (e.g., Spooler) were translated. While the translated word was in the candidates' native language, it was not recognized or understood by many. Since some candidates did understand the translated term, we could not simply revert to the English word. Our general solution was to provide the current translated version of the item and included a button on the screen labeled “English.” When pressed, this button would show the item in the original English version. Including the “original language” of the item erased any confusion and allowed candidates to complete the item without the negative influence of that particular problem. The dual-language item was new and made possible by the computer administration of the certification exams. Later, psychometric analyses indicated an improvement in item statistics coincidental with the change. That single technology innovation caused a major complaint about our translated exams to go away forever.
Recently, I have introduced and encouraged the use of an improved multiple-choice item type, called the Discrete Option Multiple Choice (DOMC).[1] Computerized testing technology allows a typical MCQ to be presented in such a way that it removes a host of problems inherent with the traditional version. Benefits in exam security, fairness, and validity, as well as accessibility, are immediately realized. The scientific research to date has confirmed many of these benefits. Recently, Kingston et al. confirmed in a research report[2] that there is no psychometric reason for not using the DOMC item type in examination programs.
There are certainly many reasons (excuses?) not to innovate or use others' innovations. I have listed a few of them above. However, I firmly believe that there is little to be gained from the status quo, much to be lost by clinging to it, and much to be gained from abandoning it. To be sure, current and yet-to-be-discovered technology solutions provide a safe and welcome light at the end of the tunnel. However, even with the best technology solutions if we continue to reject change, we will only continue to live in the past. Many creative and effective assessment solutions are already available. Only after we confront our barriers (e.g., fears, skepticism, and excuses or other barrier[s]) and reach beyond our comfort levels can we effect the kinds of change for which we aspire.
Financial support and sponsorship
Nil.
Conflicts of interest
This work is an adaptation of a blog originally appearing on Caveon's website (www. Caveon.com).
| References | | |
| 1. | |
| 2. | Kingston N, Tiemann GC, Miller HL, Foster D. An analysis of the discrete-option multiple-choice item type. Psychol Test Assess Model 2012;54:3-19.  |
|
Search Pubmed for