View all blogs

Dean’s Blog: How I dealt with my “connection anxiety” while going from paper-pencil to digital testing.

by Dr. Dean Delis

I have a confession to make. During the early years that the Pearson research team and I were developing the D-KEFS™ Advanced (all-digital tests of executive functions), I was reluctant to use digital assessments in my private practice. I felt a bit hypocritical; here we were striving to create state-of-the-art digital tests of executive functions, yet in my own practice, I was clinging to tried-and-true print versions of tests, even for questionnaires like the MMPI-3®.

Looking back, my reluctance to go digital stemmed from what I call “connection anxiety”, a deep-seated fear of losing test data to some unforeseen connectivity glitch. It’s akin to “range anxiety,” which can make prospective buyers of all-electric cars hesitant to take the plunge. Valid test data are crucial in people’s lives, guiding accurate diagnoses of developmental or acquired neurological disorders, and providing some of the most important information for IEP school assessments, disability evaluations, and criminal and civil forensic cases. My greatest concern was conducting a high-stakes neuropsychological evaluation only to lose the results from one or more tests.

Finally, I decided to tiptoe into the digital arena by starting with psychological questionnaires like the MMPI-3. I won’t lie — when I first began administering the MMPI-3 digitally, I had more than a few panic moments. One stands out: an examinee shut off the laptop halfway through the inventory. I was certain the results were gone forever. But when I restarted the laptop and reopened the questionnaire, it picked up exactly where the examinee had left off. I remember silently — and profusely — thanking the digital gods for saving the test data (and my peace of mind).

Then I had an “aha” moment: it’s just as easy — if not easier — to lose valid data with paper-pencil tests as with digital ones. Take the paper-pencil version of the MMPI-3, for example. If an examinee skips a question and gets out of sync on their true–false responses, the entire protocol can be invalidated. If they mark the paper answer sheet sloppily, it can be hard to tell which answers they intended. And if the practitioner makes a simple input error while manually entering the examinee’s responses, the findings can be thrown off. With the digital MMPI-3, these problems are almost impossible—each item appears individually, and scoring is captured automatically. That’s when it hit me: digital testing can actually reduce measurement error in a significant way.

Then it dawned on me that the same problems can happen with paper-pencil versions of cognitive tests. Take the Color–Word Interference Test (CWIT), based on the classic Stroop task. The original D-KEFS has a print version, while the new D-KEFS Advanced offers an all-digital version. In the print version, the examinee names or reads aloud rows of colors or words on a single page. If a fast responder skips an item or an entire row, it can be extremely difficult for the examiner to know exactly which responses were correct or incorrect, potentially invalidating the entire task.

The all-digital CWIT eliminates this issue. Each item appears individually on the examinee’s iPad; they select one of three response options, and only then does the next item appear. Skipped items are nearly impossible, and the system automatically captures correct responses, error types, response times, and other process data, and calculates age-referenced and age/education-referenced standardized scores. I realized once again that, for valid results, the digital CWIT is actually safer than the print version.

Here’s another example. On all print versions of trail making tests, including the one in the original D-KEFS, the examiner has to keep hawk eyes on the examinee’s line-drawing, ready to stop them the instant they make an incorrect connection. But some examinees draw the lines so quickly, they can make two or more errors in a split second before the examiner can halt them, creating a non-standardized assessment.

The new D-KEFS Advanced Trail Making Test eliminates this problem. When an examinee makes an error, the digital platform instantly displays a red line and X, prevents further responses from the incorrect circle, and only allows them to continue from the last correct circle. Once again, the digital version significantly reduces measurement error and produces a more valid assessment.

This isn’t to say that digital cognitive tests are problem-free. For instance, if an examinee’s iPad® isn’t adequately charged at the start of an assessment and runs out of battery mid-task, the data aren’t lost — the task can be resumed once power is restored — but the break in the middle makes the administration non-standardized.

To help prevent these problems, Q-Interactive® includes safeguards. For example, if either the Client’s or Practitioner’s iPad® battery drops too low, a notification appears on the practitioner’s iPad, alerting them before the device shuts down.

One last point: in the national standardization study of the D-KEFS Advanced in which the all-digital tests were administered to 1,280 children and adults (ages 8 to 90) and 160 participants from various clinical groups (e.g., ADHD, ASD, TBI), connection problems or lost data were extremely rare, occurring in only about 0.01%  of administrations.

When I’m conducting a neuropsychological evaluation in my private practice, I’ve accepted that I’ll always feel at least some anxiety about possibly losing test data, after all, I live in Southern California, and you never know when the next earthquake might hit. Still, I now feel more secure using digital versions than print versions of tests. And my next personal goal? To take the plunge and buy an all-electric car!

Best,
Dean

For more information about D-KEFS Advanced

New to Q-interactive?