View all blogs

Qualitative Data on Digital-Only Neurocognitive Tests: Feast or Famine?

by Dr. Dean Delis

In my first post in this blog series, I explained how the D-KEFS™ Advanced represents a significant departure from traditional revisions of paper-pencil cognitive tests. We went all in and developed digital-only tests that are administered and scored exclusively on iPads. Unlike the WISC-5, WAIS-5, and WMS-5, which offer both print and digital-assist formats, the new D-KEFS Advanced has no print equivalent. There are no paper stimuli, record forms, or physical manipulatives, such as the round chips and wooden pegs used in the original D-KEFS Tower Test. Instead, examinees use a stylus to interact directly with the Client iPad, which presents the test stimuli and digital manipulatives, and automatically captures and scores responses. Examiners work solely from the Practitioner iPad, which provides instructions, prompts, timing, and a real-time record of the examinee’s responses.  

Why go fully digital?  One major reason is that a digital-only format allowed us to develop more dynamic and cognitively demanding tests of multitasking and other executive functions – tasks that provide stronger clinical sensitivity, as reflected in larger effect sizes, in clinical groups such as ADHD, TBI, and other conditions (see Delis & Kaplan, 2025). These enhancements would not have been possible if we also had to create equivalent print versions. 

But in striving for greater clinical sensitivity, did we risk losing one of the hallmarks of cognitive assessment – the ability to observe and interpret the qualitative aspects of performance? Understandably, many psychologists are hesitant to adopt all-digital tests out of a concern that these rich sources of behavioral data might be lost. And to be fair, some existing digital cognitive tests do fail to capture essential qualitative features of performance. 

How important are qualitative data, especially on executive function tests? Consider the print version of the original D-KEFS Tower Test, which uses wooden chips of varying sizes that have to be moved across wooden pegs. The examinee’s task is to arrange the chips into a specified configuration while following several rules: use the fewest number of moves possible, move only one chip at a time, and never place a larger chip on a smaller chip. On this task, some of the most meaningful clinical data comes not only from the final arrangement, but also from the way the examinee qualitatively interacts with the manipulatives.

For example, an examinee might lift one chip with one hand, hold it in the air (or—as one child I tested once did—hide it behind his back), and then use the other hand to move and place a second chip. This response is more than a basic rule violation error (failing to move only one chip at a time). It also provides a vivid qualitative marker of poor spatial planning—an impulsive “easy way out” that bypasses the intended cognitive demands of the task. 

So, the question naturally arises: does the new, all-digital D-KEFS Advanced Tower Test sacrifice the ability to observe these kinds of important qualitative behaviors?

If there is one lesson I learned from my mentor and close friend, the late Edith Kaplan, it’s this: never let meaningful qualitative data slip by the wayside. Edith would have never forgiven me if I had designed a cognitive test that sacrificed these essential qualitative features in pursuit of greater clinical sensitivity. But what if we could have both? What if we could create all-digital tests that not only offer stronger clinical sensitivity, but also automatically capture, score, and even provide normative data for a wide range of qualitative behaviors?  That was precisely one of our central goals in developing the D-KEFS Advanced.

Consider again the new D-KEFS Advanced Tower Test. We designed this all-digital task not only to permit the same behavioral response styles that occur on the physical version, but also to automatically capture, quantify, and provide normative data on those behaviors. For example, on the new version, the examinee can use the stylus to lift a chip off its starting table (we now use tables instead of pegs, which makes the task easier for patients with motor difficulties), leave the chip suspended in the air or off to the side, and then use the stylus to lift and move a second chip. Whenever this type of response occurs, the digital platform instantly and precisely performs all of the administrative tasks the examiner is supposed to do. It immediately provides the examinee with error feedback (an error tone, a shaking-chip motion, and a red X over the chip), returns the chip to its prior location, and—most importantly—records and scores the response as a two-chip rule violation error, ensuring this important qualitative behavior is captured. 

In both the original and new Tower Tests, the examiner is required to provide a specific prompt the first time the examinee commits each type of rule violation error.  On the original version, this can be difficult for examiners to track, particularly for examinees who move the chips rapidly while the examiner is frantically counting moves. In the all-digital version, however, every correct and incorrect move is automatically captured and scored by the platform. And when a prompt is required, the system temporarily pauses the Client iPad so the examinee cannot make further responses, provides a notification on the Practitioner iPad that a prompt needs to be read and what to say, and then activates the Client iPad after the examiner touches ok. The automation of the all-digital version relieves examiners of the heavy administration and scoring demands of the print version, giving them more freedom to observe important qualitative features of performance while also reducing measurement error (e.g., miscounting moves or errors). In fact, the D-KEFS Advanced Tower Test is so easy to administer, it allows examiners to carefully observe nuances of the examinee’s behaviors, such as brief moments of distraction, frustration, tearful reactions, pain behaviors, or self-talk strategies.

Developing an all-digital version also allowed us to capture and automatically score several other qualitative features that, while observable on the print version, are extremely difficult for examiners to record – mainly because they are often so busy counting moves and catching errors. For example, an examinee might lift a chip from its table, place it on another table, and then return it to the original table. We call this a “1-Back Response,” which reflects a trial-and-error strategy rather than effective spatial planning (it’s an unproductive move because it runs counter to the task’s goal of using the fewest number of moves possible). Other trial-and-error responses include “0-Back Responses” (lifting a chip, holding it in the air, and then placing it back on the same table), and “2-Back Responses” (lifting a chip, moving it to one table, then a second table, and finally returning it to the original table). On the physical version of the Tower Test, it’s nearly impossible for examiners to reliably identify and count these important qualitative response types while juggling all of the other administrative demands of the task. By contrast, the D-KEFS Advanced Tower Test automatically captures and scores these important qualitative responses and provides normative data for them.

It took us years to develop the D-KEFS Advanced—much longer than we originally anticipated. One reason is that we were relentless in our pursuit of creating the next generation of all-digital neurocognitive tests that allow the examiner to feast on more qualitative and quantitative data than they ever imagined. Edith wouldn’t have wanted it any other way.

Best,
Dean

For more information about D-KEFS Advanced

New to Q-interactive?