Overview

A study was conducted to determine what aspects of an app demo would help customers better understand a product feature and draw them to download the app.

Challenge

I was assigned to support Kelley Parsons, the Chief Researcher, with a study regarding app demos. The stakeholders wanted to test how three different demos impact consumers’ understanding of a specific app feature and their desire to download an app. 

There were several challenges to this study:

The stakeholders involved in this study had not yet worked with my team and were not familiar with user research. Therefore, it was a challenge for them to understand the research process and establish specific goals. There was only a limited amount of time to gather feedback when participants had to experience all three prototypes within the span of an hour and a half. Although the stakeholders wanted to glean a lot of information from the study, we had to establish the most important goals and make sure the study remained in scope.  

Another challenge was the deadline. As the Chief Researcher, Kelley did not have a lot of time to spend preparing for this study. Furthermore, she was going on vacation the week after the study was scheduled to run. Therefore, there was no opportunity for any delays in the study. 

Lastly, the prototypes required highly advanced techniques and programming. Though significant amount of progress had been made, it was uncertain what state the prototypes would be in given the tight deadline and effort it would take to create the prototypes.

Process

Identifying the Known Information

Although there were still some unknowns with the study, there were aspects that were established so that I could begin preparations. 

The overall structure of the study was already defined: 

• Each session would have 3 participants

• After each participant experienced a demo, they would take a survey and then have a group discussion on their experience

• Data collection would only occur in one day

• Kelley and I were the only dedicated researchers to the study

• Stakeholders were interested in viewing the sessions

• Participant requirements were established

Coordinating Materials & Study Space

I coordinated with stakeholders to ensure that all the materials we needed were gathered, which consisted of: 

• At least 3 hardware prototypes

• All the app prototypes on 3 different phones

• 3 iPads with keyboards 

• Chargers for all of the technological equipment

• Video equipment

For the study space. I picked an appropriate conference room that allowed room for the participants to interact with the demos individually. Kelley would be able to project her slides to facilitate the group discussions, and the stakeholders unobtrusively sat in the back of the room to remain undetected by the participants. 

In regards to the app prototypes, I monitored their progress by receiving daily updates. In the meantime, I used the videos depicting the ideal flow of the prototypes in order to familiarize myself with the prototype before data collection.

Technological Setup 

I used the logic within the Qualtrics survey platform to counterbalance the 3 demo surveys as well as create a note-taking tool within one project. As a result, the data exported into one CSV file that clearly divided the data by demo for quick and easy data analysis. I created the template for the survey while Kelley worked on the actual survey contents. 

To monitor the participants’ progress through the demo, I discovered a way to share the phone screens on the iPad using Microsoft Teams. With the three participants spread out, Kelley and I would be able to watch al three participants walk through the demo from afar using the iPads. That way, we would be able to step in if any issues arose.

Recruiting Participants

With the recruitment criteria known, I pre-screened participants from the User Research Participant panel. I then created the study screener, sent it out, and scheduled the eligible participants.

Outcomes

The data collection ran smoothly. In the end, the technologist and one of the coordinators I was working closely with assisted with monitoring the participants’ process through the app demos. Therefore, the phone screen sharing was not needed. However, this method of viewing phone screen would be useful for future studies. 

Kelley presented the findings to a smaller team before having a larger readout that included the Director of Product and Communication Design at Bose. 

We were able to provide a lot of insights and recommendations on the overall flow of the demo, aspects of the demos that resonated with the participants and ways to draw users to download the app. 

The stakeholders developed another prototype based on these recommendations and conducted a follow-up study.