Scalable Usability Testing
The teams kept testing after I stepped back
The team assumed pilots would want dollar amounts first. In my first usability round, 3 of 4 pilots were not concerned that dollars were missing. I turned that round into a toolkit that Delta’s product teams ran on their own.
Results at a glance
The toolkit
With these three documents, anyone on the team could run a round and get results we could compare.
Facilitator guide
A fixed script, so a product manager or Technical Product Engineer could run sessions without me.
SUS scoring workbook
The same scale in every round, so scores could be compared across iterations and teams.
AI-assisted summary
AI wrote a first draft of the findings, and the person who ran the sessions checked it.
Problem & stakes
Feedback came late, and we could not compare one version with the next.
The Crew and Catering teams got their feedback informally, and it varied from project to project. User needs often came up after development had started, and nobody could tell whether a new version was better than the last one.
I was the only UX resource across several squads, so testing could not depend on me running every session.
Process
I built the toolkit on a live project and then handed it to the team.
Running the first round
In January 2024, I tested a crew pay application with four pilots: two chief pilots and two assistant chief pilots. The team assumed pilots would want dollar amounts front and center. Three of the four were not concerned that dollars were not shown, because they think about a reroute in terms of their schedule. All four wanted the rerouted leg to stand out against the original. So later phases of the design compared the original schedule with the actual one and added pay on top.
Documenting it as a toolkit
I wrote up the round as a facilitator guide, an SUS scoring workbook, and a summary template, so later rounds could be compared no matter who ran them.
Handing it to the team
A product manager and a Technical Product Engineer ran the next two rounds using my documents.
Trade-offs
Small rounds were fast, but they cannot prove a statistical difference.
Each round had four or five participants. That is enough to find usability problems and set a direction, but it cannot prove a statistical difference. So I report findings as counts (3 of 4 pilots) and do not turn them into percentages.
AI drafted the summaries. The person who ran the sessions checked every summary against what they had observed.
A fixed script and the SUS scale made rounds comparable across teams. The cost was that facilitators had less freedom to adjust a session as it went.
Outcome
The teams continued testing after I stepped back.
Crew pay application: the SUS score was 71.3 in my round, 91.3 in the second round, and 90.2 in the third. The design team raised the score through iteration. My part was the research that set the direction and the toolkit that kept measuring it. Pilot feedback also led the Crew team to prioritize a mobile approach.
Catering application: the product manager had planned to let reservation agents search for passengers by booking reference (PNR) or SkyMiles number when assigning pre-selected meals. Testing showed that agents searched only by PNR. The SkyMiles option was removed, which reduced the estimated build effort by 50%.
Adoption: the Catering team adopted the toolkit, and In-Flight Services adopted its quantitative approach for flight attendant focus groups.
Participant experience: 94.2% satisfaction in a two-question survey I ran after each session, covering the facilitation and the tools used.
“Watching you lead usability testing for [crew pay] was insightful. I am planning to continue to incorporate usability testing with the crew team for future initiatives.”
— Stephen Young, Product Manager
Reflection
I would set up a single shared location for findings from the start. Results were kept in separate summaries, which made it difficult to see patterns across projects.