Findings from an exploratory user research study highlighting the unique insights and practical UX recommendations shared by participants with cognitive disabilities.
In the summer of 2024, I became co-chair of a working group of expert researchers who came together to determine how best to perform accessibility testing with people with cognitive disabilities. This was work I did for Fable, where I am currently VP of Innovation.
Cognitive disability is an umbrella term for several disabilities that impact how people process information, and it usually affects memory, focus, and/or learning. It is the most prevalent disability in the U.S. (13.9% via CDC), and cognitive disability is increasing rapidly (Yale study).
We set four goals for ourselves to learn how to work with this audience:
- How should we recruit and screen participants?
- What are best practices for research with cognitive participants?
- Do these methods work in a real study?
- Documenting what we learned so that we could share it.
We created a screener to recruit people who self-identified as having challenges with memory, focus, and learning. We also reviewed published studies that involved cognitive testers to learn best practices for working with them.
Next, we tested these best practices with an initial group of 25 testers in a pilot study. We fine-tuned our approach iteratively and created a guide to running user interviews with cognitive testers and a survey that could quantify their experiences using digital products. Finally, we documented what we learned.
After our pilot study with this new group of testers finished, I felt that they would uncover more usability insights than the general population (gen pop) user research participants I’d worked with in the past. I set out to validate this hunch.
The Cognitive Usability Study
I decided to run a joint study with Fable’s partners at the University of California, Irvine, in collaboration with Syed Fatiul Huq and with help from Fable researchers Pranav Pidathala, Ali Brown, and Michael Fagan to see if my hypothesis about finding more insights with cognitive testers proved true or not.
I generated three websites for the study using an AI prototyping tool. I wanted three different types of sites with different user goals and content so I could test a variety of tasks in the study.
Table 1: Websites and Tasks Tested
| Strong Snacks | Turning Pages | Crown & Comb | |
|---|---|---|---|
| Description | A website for three-ingredient high-protein recipes. Recipes can be browsed by category (vegan, muscle building, etc.). The site also features blog posts about protein and contact information. | A website for a bookstore with a catalog of curated reads. It features extensive filtering by book genre, a book swiping feature to build a profile of likes and dislikes, custom book lists, a shopping cart, and checkout. | A website for a hair salon that allows you to book appointments and consultations online. It has a VIP program and a variety of special packages visitors can buy. |
| Design | Simple, brutalist, bright, lots of pictures. | Moody, classic, dark, lots of pictures of book covers. | Bold, clean, black and white with bursts of color. |
| Content | Recipes, blog posts. | Books and book lists. | Services, experience guide, membership information. |
| Key functionality | Filter by category, newsletter subscription. | Shopping cart, book matching, book lists, recommendations. | Appointment booking. |
| Tasks | 1. Find a recipe for a high-protein snack. 2. Find a blog about protein and read it. 3. Find a way to be notified about new recipes and blog posts. | 1. Find the book swiping feature and use it on 10 books. 2. Find the recommended book list. 3. Add books from two genres of your choice to cart. 4. Checkout the books in your cart. | 1. Find the prices for getting a haircut. 2. Book a haircut appointment. 3. Find the price for the bridal package. |
We used a single screener with questions about memory, focus, and learning, and screened participants into two groups based on whether they self-identified as having cognitive challenges or not.
Cognitive disability includes neurodiversity. Neurodivergent is an umbrella term used to describe people whose brains process information and learn differently. It is most commonly used for people who have learning disabilities (e.g., Dyslexia), ADHD, and Autism.
We ran 30 user interviews, 10 per website, with an even 5/5 split between cognitive and gen pop participants for each website. In each session, a participant completed all the tasks for one website during an online user interview facilitated by one of the researchers involved in the study.
All participants completed an Accessible Usability Scale (AUS) survey at the end of their session. This is a free, Creative Commons-licensed 10-question survey to evaluate the usability of websites and mobile apps.
Data Analysis Approach
I reviewed all the study recordings and transcripts and made note of every time a participant raised a concern, question, or difficulty, or asked a question about how something worked. I counted all of these as issues. I also noted where a participant missed something that was part of a task, even if they didn’t notice it themselves, and I recorded every suggestion for improvement made by participants.
Examples of issues found included:
- Photo is too tall and requires a lot of scrolling to get to content (noted by participant).
- I get no feedback when I like or dislike a book (noted by participant).
- Participant missed the required P.O. Box checkbox the first time (observed by me).
Examples of suggestions included:
- I would like to see a protein comparison in a table.
- The “More information” tab should be moved up higher.
- I would like more information on how the recommendation list is created.
Issues and suggestions were counted once per participant, even if they mentioned the same thing twice, but there are, of course, repeat issues and suggestions across the different participants. It is expected in UX research with multiple participants that you’ll find similar issues with each participant, and that is a signal that an issue is a universal challenge.
Findings of the Cognitive Usability Study
Across the three websites tested:
- Cognitive participants identified 197 issues.
- Gen pop participants identified 113 issues.
- Cognitive participants made 93 suggestions.
- Gen pop participants made 54 suggestions.
- Cognitive participants surfaced more issues related to content, buttons, icons, visual elements, and media than gen pop participants.
The results aligned with my instincts: participants with cognitive disabilities identified 1.8 times more issues and made 1.8 times more suggestions than gen pop participants.
Let’s dive deeper into the data for each website. Note that an AUS score ranges from 0 to 100, with higher numbers representing better usability than lower numbers.
Table 2: Strong Snacks
This site had the simplest design and content of all websites tested in the study and accordingly had the lowest overall issues and the highest median AUS scores. The data aligns with what you’d expect from an easy-to-use and simple website.
On this website, cognitive participants found 3.4 more issues and made 2.2 more suggestions on average. Their average score of the overall experience was 13.7 points lower than that of the gen pop participants.
| Total issues | Average issues | Median issues | Total suggestions | Average suggestions | Median suggestions | Average AUS | Median AUS | |
|---|---|---|---|---|---|---|---|---|
| Gen pop | 32 | 6.4 | 6 | 13 | 2.6 | 2 | 90.5 | 97.5 |
| Cognitive | 49 | 9.8 | 9 | 24 | 4.8 | 4 | 76.8 | 73.0 |
Table 3: Turning Pages
This was the website with the most varied functionality and the most tasks to complete (4), so it’s not surprising that participants found the most issues.
Here, cognitive participants found 6 more issues and made 3.2 more suggestions on average. They also scored the overall experience 17.2 points lower than gen pop participants on average.
| Total issues | Average issues | Median issues | Total suggestions | Average suggestions | Median suggestions | Average AUS | Median AUS | |
|---|---|---|---|---|---|---|---|---|
| Gen pop | 55 | 11 | 10 | 26 | 5.2 | 4 | 78.0 | 80.0 |
| Cognitive | 86 | 17 | 15 | 42 | 8.4 | 6 | 60.8 | 58.0 |
Table 4: Crown & Comb
This website was intentionally designed to be complex, and task 3 — finding the bridal package — was meant to be extremely difficult to complete.
On this last website, cognitive participants on average found 7 more issues and made 2.4 more suggestions. Their average score for the overall experience was 14.3 points higher than the gen pop participants.
| Total issues | Average issues | Median issues | Total suggestions | Average suggestions | Median suggestions | Average AUS | Median AUS | |
|---|---|---|---|---|---|---|---|---|
| Gen pop | 26 | 5 | 4 | 15 | 3 | 3 | 49.5 | 35.0 |
| Cognitive | 62 | 12 | 11 | 27 | 5.4 | 2 | 63.8 | 68.0 |
Something interesting happened with the AUS scores for cognitive and gen pop participants in Tables 3 and 4. Cognitive participants scored Crown & Comb higher than Turning Pages, but gen pop participants scored the opposite — higher for Turning Pages and lower for Crown & Comb. The most likely explanation is that finding more issues on Turning Pages affected how cognitive participants rated that overall experience, whereas gen pop participants, having identified fewer issues overall, may have felt the friction on Crown & Comb more acutely. This divergence underscores how meaningfully different the two groups’ experiences of the same interfaces can be, and why including cognitive participants in usability research produces a more complete picture of where a product succeeds and where it falls short.
