How we conducted user research for our new UX research platform

Continuous Discovery Leads To Continuous Improvements

At ConnectFive, we believe the knowledge and insights that help fuel product or service innovation comes from having continuous conversations with users. In fact, this belief is so strong we designed and developed our own UX research platform in order to help our team be more efficient and effective at their jobs for our clients.

As time went on, we talked with other UX, marketing and development professionals that valued continuous user research and identified they were experiencing similar pain points. So we decided to modify our internal tool and release Handrail to the world.

Below is a step by step explanation of how we conducted one of our early research study efforts and the rationale behind each stage. Hopefully it will inspire you to have conversations and gather insights that fuels innovation too!

Research Study Purpose & Methods

In order for us to conduct good research, we needed to have a clear understanding of what we were trying to learn and how we were going to use this information. So, we developed several goals, prioritized them, and then clarified our primary and secondary goals.

Primary Goal

Although we had been conducting user research throughout our initial design and development process, we knew it was critical to keep updating our knowledge and understanding. Because, while goals and needs may remain stable, user expectations are evolving day by day. Here’s what eventually landed on as our primary goal:

• Validate/understand our target market, users, and their needs

Additional Goals

We also wanted to gather initial impressions of Handrail since it had recently launched into the market place. We felt it necessary to create additional goals to:

• Confirm and enhance our understanding of various user research practices
• Gather first impression feedback of new UX research tool
• Inform and prioritize items for our next design sprint

Research Methods

To achieve these goals, we decided to use a mixed approach of Quantitative and Qualitative research methods. The quantitative research would reveal what areas we need to focus on, and the qualitative research would help us understand context and the “why” behind the answers.

Quantitative Research: “What” do we need to focus on?

For our quantitative research phase, we decided to create and use a survey questionnaire. Surveys are easy to put together, can give you great information, but they also have drawbacks as well. One of the big drawbacks is being irritating if they get too long, so we planned to make this as quick and painless for respondents as possible. (see Nielsen Norman’s recommendation: https://www.nngroup.com/articles/keep-online-surveys-short)

We decided to limit our questions to eight in order to keep it to about five minutes to complete. Here are the topics we created, the questions we asked, and the rationale behind them.

Survey Topics, Questions & Results

Topic: Participant Background

Our first objective was to capture who was filling out this survey and what they typically did when conducting moderated user research. Knowing this would help us understand the different types of participants, how they defined themselves, as well as what they identified as important when doing research. Here are the three questions we used to help us understand these objectives and the results:

Questions

• What is the best description of your current role at your organization?
• What types of qualitative or moderated user research methods do you use?
• Which activities do you use during moderated research?

Results

(58% people indicated having more than one role)

Topic: Pain Points

We wanted to understand the pain points they experienced as individuals conducting moderated research as well as what made it difficult as an organization to conduct research. The first two questions we chose were pretty straight forward, but were a little leading and potentially bias since we used the term “pain points”.

For the third question we decided to use a re-framing technique. We believed re-framing the question as a “prevents” instead of a “pain point” could give us another angle to understand the difficulties they face.

Questions

• As an individual, what are your biggest pain points when conducting moderated research?
• What are your organization’s biggest pain points when conducting moderated research?
• What prevents you (or your organization) from conducting more moderated research?

Results

Topic: Software / Tool

We needed to know what they use to help them perform their job. Knowing this information would help us better understand our competition as well as potential considerations for future integrations. Here are the two questions we used:

Questions

• What software applications or services do you use when conducting moderated research?
• What other tools or equipment do you use when conducting moderated research?

Results

Topic: Interest in new moderated UX tool

This final topic was critical for us since it needed to accomplish two goals; pique interest in the new user research platform as well as gather leads for our next round of research. Here’s what we used:

We are currently developing a new moderated research tool. If you are interested in participating in future studies, please provide your email address below. Your email address will be used only for research purpose.

Results

This last part was not required on the survey, yet 75% of people that took it provided their email addresses. Many of them even wrote down short comments with their email addresses, such as “I am curious what pain points other people have.”, and “I’m really interested in seeing what you are working on!”.

Distribution Challenge

You can create the best survey questionnaire in the world but if you don’t get enough of the right people to take it, it can seem like your efforts are totally wasted. We learned this lesson early on and knew the two challenges we had to overcome were:

• Finding the right audience
• Getting enough of them to take the survey to feel confident in our results

For us, these challenges seemed like a numbers game. We thought if we utilized distribution channels that contained high amounts of quality people, we would be able to quickly get the numbers we needed to feel confident in our results. So, we started to brainstorm channels we believed had the numbers and the quality we were looking for.

Channels

Fortunately identifying the distribution channels was easy. Over the years we have:

• Been involved with various design and development organizations.
• Joined and contributed to multiple Slack channels and Meetup groups that relate to our profession.
• Attended many conferences where we’ve created relationships with like minded professionals.

These activities have given us the ability to tap into the exact audience we are wanting, as well as not come off as “spammy” when asking for their help.

Message

The real hurdle for us was crafting a message that would entice participants to take the survey. It needed to be short, to the point and professional. After a few rounds of revisions, here’s what we sent:

My name is Mat and I am a UX Researcher with ConnectFive (an innovation and design studio in Coralville, Iowa).

We are conducting a research study to help inform a new application we are creating to help with Moderated User Research; user interviews, concept tests, usability assessments.

If you conduct 1 on 1 research (in person or remote) and have a few minutes, we would love your help and your insights: (link to survey)

Thanks!

This simple message helped us received 67 responses in just a few days.

Qualitative Research: Getting to “Why”

At this point, we had enough high-level information to help inform “what” we might focus on. But only having that type of information is like saying: “We need to go west in order to get to California”. Having a general direction is important, but understanding context, reasoning and the details of “why” is critical when trying to solve a problem.

Recruitment

We were fortunate enough to have 50 out of the 67 survey participants provide their email addresses indicating they would like to do a follow-up interview. So, we reached out to those interested asking them to sign up for a time to speak with us more in detail regarding their survey answers.

To make the scheduling process as easy as possible, we allowed participants to choose a time in our schedule that worked for them via Calendly. Using this tool cut down on the typical back-and-forth email exchanges of finding times that works for both parties. Ultimately this lead to 30% of respondents who provided their email addresses signing up for an interview session.

We also understood that availability can change, so we sent reminder emails 24 hours in advance of each session to help avoid the dreaded no-show. Out of a total of 16 people who signed up, we encountered only one no-show (who eventually rescheduled).

Here are the interview guide topics and probing points we used for the interview and the rationale behind them:

Part 1: Survey Follow-Up Discussion (30 min)

Topic: Participant Background & Role

Probing points

• Confirm and clarify participant background, their role, and responsibilities
• Dig into and understand why individual and organizational pain points are a problem for them
• Have them explain and walk through their specific research process
• Have them talk about the tools and methodologies they use and why they use them

Results

Where do they reside?

(Chart created with Handrail)

What do they do?

(Chart created with Handrail)

• Findings
  • They have various backgrounds, but majority of them do similar activities
    • Research
    • Product & Design testing
  • 75% Using Google Doc and Forms for moderated research, but 25% Said “Not an Ideal Format” & “Cumbersome and Clumsy”

What Do They Need/Want?

(Chart created with Handrail)

• Findings
  • Want to save time
  • They use 5 or more tools to help them gather information about their users and the product
  • The desire for more team involvement and company support

• Action Items
  • Explore additional ways to help with:
    • Analysis
    • Collaboration

Part 2: Handrail Discussion (30 min)

Topic: Demonstration

• Quickly prepare your research effort
• Easily capture conversations and notes from participants sessions
• Analyze the data captured during conversations and create insights
• How it supports collaborating with your team in order to make more informed design and development decisions

Topic: Initial impressions and participant feedback

Probing Questions:

• What are your initial thoughts on what you’ve seen so far?
  • Probe for positive/negative/neutral thoughts
• Do you have any questions on what you’ve seen so far?
  • Probe for areas of confusion/things expected but missing
• What would make Handrail more valuable to you?
  • Probe for opinions on potential improvements
• If you could wave a magic wand and have Handrail do anything, what would it do?
  • Probe for opinions on abilities if there were no imposed restrictions

Results

(Chart created with Handrail)

Positive Reaction Highlights

• Team Collaboration
• Question Template:
  • Time Saving
• Before Seeing Pricing Page:
  • “Looks Rich, Looks Expensive” “This seems like and Adobe License Level”
• After Seeing Pricing Page:
  • “Wow, I don’t want to say it out loud, because you might change it…but this is a good deal!” “I pay $15 for Spotify, I would pay twice this to make my job easier!”

Negative Reaction Highlights

• Export Results:
  • Wanted various types of exported data beyond CSV
• Findings Format:
  • Wanted more automated analysis info from Results
• Reporting:
  • Wanted to convert results into a presentation format that could be used for clients

Wrap Up

After we analyzed all the data and had discussions as a team, we were able to clearly identify additional items for design exploration as well as areas we need to explore for our next round of research.

We also preformed a retrospective on the entire process. Here are the top take a-ways:

Was the research effort a success?

Yes. Absolutely. In our opinion, every research effort is a success because you learn something new each time. In this specific case, we achieved our goals and smoothly integrate our findings into a design sprint.

What would you do differently next time?

Involve additional team members in the process. Our team practices continuous development and delivery methods so communication and shared understanding is really important. We believe the less distance everyone has between understanding the user, the better the overall product is going to be.

Take-A-Ways

Continuous discovery leads to continuous improvements

UX researchers conduct a customer discovery and develop personas in the early product development stage, but many UX researchers use the early stage of information forever and never updating them. Things are changing, even faster in the digital world. Keeping updating your awareness about users will lead to the right direction for your product improvement.

Everyone can and should do user research

User research has historically been considered an activity that is expensive, difficult, time consuming, and should only be done by experienced professionals. However, it does not have to be that way! Having a strong plan and using proper tools when conducting user research can provide efficient procedures and high-quality results.

About Handrail

We built Handrail to help teams collaborate throughout the entire user research process. Plan, collect, analyze, store, and share your research all in one location. Sign up for a free 30-day trial today.

• Start Free Trial

Co-authored by: