Benchmarking Longitudinal Survey

Solo UX Researcher | 13 weeks

THE PROBLEM

The Beam Design System at Viasat had existed for various stakeholders for the part 3.5 years. With it came an entirely new structure for producing designs for Viasat. This meant a change of processes and new considerations for product teams. The Beam team was now asking important questions about user experience and user satisfaction during Beam’s use and wanted to understand high-level issues to probe into during major version releases.

RESEARCH AND RESULTS

A longitudinal survey was built as a quantitative benchmarking study. It intended to-

  • Be used for every major version release, reaching 300+ users across 5 stakeholder groups that interact with the Beam Design System

  • Benchmark experience across all touch-points of Beam with users using UX metrics

Through this first evaluative study, we were able to call attention to improve 3 out of 5 KPI for Beam- adoption of Beam, feature release frequency, and stakeholder engagement.

DEEP DIVE INTO THE ‘HOW’

I. INFORMATIONAL INTERVIEWS

To clarify the problem area, understand timelines, and expectations, I scheduled 6 interviews with the Beam Team to understand in depth-

Who interacted with Beam and how

The Beam design system consists of reusable components and guidelines to provide Viasat’s digital products a consistent visual language and a set of standards. It consists of elements that promise usability, standardisation, and a consistent brand identity. Additionally, it promises accessibility and velocity from design to development. It is currently being used across the company for products by-

UI/UX

Designers

Use tokens and design components to create wireframes

What a successful interaction looked like

</>

Developers

Use the component library to convert designs to code

Frustrations for each stakeholder

Product Managers

Product visionaries that lead teams using Beam

Product Owners

Technical owners that lead teams using Beam

II. SERVICE MAP

At this point, a high-level service map was a great design artefact to visualise the design system’s components and how all stakeholders interact with it.

III. DRAFTING A LONGITUDINAL STUDY

Why a survey?

  • Great research method to reach 300 users and appropriately assess user experience for different stakeholders using custom logics

  • Valuable quantitative insights and is cost and time-efficient

  • Enables benchmarking by comparing past years’ experience, hence helping the Beam team move closer to their mission of promoting brand consistency, usability, and accessibility, and assisting in the production of better designs

The implications of results would look different for different users due to different interactions with Beam and different measures of satisfaction. For example, more design components would mean success for designers, and for product owners, it may be increased consistency across the designs of their products.

With this clarity and context, I started my process of drafting questions on Figjam for the 4 stakeholder groups. For each of the stakeholders, I targeted all points of interaction with Beam-

IV. FEEDBACK LOOPS

After completing my survey draft, I engaged in iterative feedback cycles with 2 senior researchers. The aim was to ensure the study was-

  1. A robust quantitative research study, which meant checking for good research practices like not asking leading questions, ensuring clarity of language, balancing survey fatigue with amount of feedback, among others.

  2. A good longitudinal study that can be used for the upcoming major version releases, which meant asking questions that allow for comparison of findings across versions to understand how the metrics have changed over time.

V. SURVEY DRAFT IN QUALTRICS

Once we were all satisfied with the quality of the survey, I presented it to the Beam team to ensure there were no topics left that they wanted user feedback on. The survey was then coded in Qualtrics, a great platform to implement complex logics like routing participants to the appropriate survey flows according to role. After conducting 3 pilot sessions with the Beam team, and incorporating minor changes in the survey, the study sent to all stakeholders over email and Slack (Beam Support channels).

VI. ANALYSIS AND TELLING THE STORY

The survey results came in after my internship ended. It received 49 responses- 23 developers, 20 designers, and 6 Product Managers and Product Owners

Overall, the satisfaction and confidence in using Beam are high across all stakeholders

All stakeholders considered the onboarding experience to be excellent and great

However, most developers mentioned they did not know the ‘Way of Working’ document was provided to them to support onboarding

Most developers never and designers rarely make contributions to Beam due to it not being prioritized by the product lead, the design system being sufficient for their use, or not knowing how to contribute

Most Product Managers and Product Owners sometimes or rarely prioritize Beam version updates in their sprint planning since it is not a priority, they need guidance, they rely on others to do it or it is too confusing to understand

All stakeholders never or rarely participate in Beam Office hours due to not being invited, not being available, or being busy

IMPACT OF MY WORK

  1. These insights provided the Beam Team with calls to action on their current procedures, especially onboardin and feedback, allowing them to improve 3/5 KPI in the use of Beam.

  2. As the first-ever longitudinal study for the design system, this survey now provides a way for the Beam Team to track metrics over time across different interaction points. They can measure user satisfaction and confidence among other parameters, and compare these across version releases to track their progress.

MY LEARNINGS

  1. Design Systems are COMPLEX! Working with the Beam Design System gave me first-hand exposure on learning how to juggle the needs of different stakeholders- Beam team, Researchers, designers, developers, product managers, and product owners!

  2. I learned that there are fundamental differences in designing a regular study vs a longitudinal study. Longitudinal studies are meant to be replicated. There need to be quantitative measurements that can be compared year after year. And so, they need to be catered to carefully while designing.

  3. Learning from collaboration was one of my key takeaways. Not only was one of my main learnings through working with the Beam Team in understanding design systems, but I also gained valuable insights by collaborating with senior researchers.

  4. Trust the iterative design research process. In the middle of my iterative feedback cycles, there was a point where everything felt wrong. However, it is important to step back, take a breath, and get back to the process with a lot of trust. As one of my seniors very accurately stated,

‘UX Research is a little bit of science and a lot of art’