AR Accessible Glasses

Accessible AR Glasses

Smart AR glasses experience for the hard-of-hearing to engage in social interactions

User

People with Hearing loss

Time

3 Months

Team

Amy Tang Fumi Tsukiyama Joyce Tsui

My Focus

Co-Design
Prototyping

Overview

This accessibility project demonstrated my co-design process with a design partner experiencing hearing loss. It showcased how I design with, not for, people with a disability. We collaboratively arrived at a solution involving accessible AR glasses with open captioning.

The project highlights an inclusive design process, which included semi-structured interviews, the “hot potatoes” co-design method, and usability testing. Some prototyping methods included body-storming, Figma digital prototyping, and rapid paper prototyping.

My role was a UX designer, collaborating closely with 3 teammates and the design partner, Ryan.

A Story of Our Restaurant Customer: Ryan

One day, a customer named Ryan walked into our restaurant with four servers (designers): Amy, Fumi, Joyce, and I. He was wearing hearing aids, and a safety helmet for construction management, and was using an app called WIDEX to adjust the settings of his hearing aid.

We had many assumptions about his allergic (challenges):

Social interaction

There may be embarrassment/shame from asking for hearing accommodations

Recreation

He might feel limitations for entertainment: Disneyland, movies, etc.

Safety at work

Working on the construction site may be unsafe as he cannot hear warnings from coworkers

Understanding Ryan’s Needs

As servers, we first understood Ryan’s actual serving needs with a semi-structured interview and confirmed our first two allergy assumptions. However, what he was concerned about the most was NOT safety at work but the level of engagement in social interaction. Together with Ryan, we identified 3 key goals (tasks).

Identify who is speaking in social interactions

Ryan relies on lip reading and additional cues such as body language and visual representation

Provide captioning

Ryan cannot lip-reading when someone is not facing toward him

Send accommodations requests

Ryan cares about not breaking the flow or vibe of conversations

Cooking Instructions (Design Requirements)

Affordable & Simple

Privacy & Transparency

Design sparks “Joy”

Multiple Sources of Cues

Creating 2 Main Course Options with Cookware

Based on Ryan’s needs, the kitchen used storyboards, Figma low-fidelity prototypes, and rapid paper prototypes to come up with 2 cooking (design) ideas for the main course:

Microphone Kickstand

Smart AR glasses

We used body-storming to prototype the experience enabled us to test the preliminary concept design of smart glasses and showed a Figma prototype to demonstrate the digital experience added to the existing hearing aid app.

Body-storming

Figma prototype

Ryan Joined to Cook the “Hot Potatoes”

During the cooking (co-design) session, we passed around the “hot potatoes”, which were the 2 cooking (design) ideas, among Ryan and different cooks (designers) to build on each other’s ideas. Involving Ryan to be actively part of the cooking process helped us generate valuable feedback about key features we should include and unexpected findings.

Unexpected Findings

Biggest surprise: how Ryan utilized captioning

Listening to a conversation is an intensive task as he has to pay attention to lipreading, body language, and much more. Thus, he only glances at captioning to quickly understand the main ideas rather than relying on it for word-by-word accuracy. It is a priority for him to keep up with the conversation.

Tasting Feedback

In the final tasting session with our customer, Ryan, we reviewed all the cooking (design) decisions and workflows. Based on his feedback, our solutions successfully addressed the problem of identifying who is speaking in social interactions and asking for hearing accommodations during conversations.

Simple and clear design

There was never a moment where Ryan expressed confusion or thought something was not intuitive

#1 Important Task: Identifying who is speaking

Matching the speaker border highlight color with the color of the captions is more important than displaying the name of the speaker in the captions

Scenario gaps remain

Gaps remain because we haven’t accounted for all possible use cases and scenarios

Reflections

Designing with, not for, hard of hearing community

With participatory design, we generated valuable insights and feedback during ideation and key feature prototyping

More protocol planning and prototype preparation

I didn’t give clear directions to the facilitator when to point to the slides, and that caused some confusion for Ryan as to when to look at the computer screen and lipread

Exclusion consequence for people with visual impairments

Adding space to have a customized accommodation request message may make it be more inclusive of different kinds of disabilities