Questions for Questions: How we surveyed our users about our new tool

It has been a couple of months now since we publicly launched Questions, our new online survey tool that complements all the other UX research you’re doing with Optimal Workshop. Since launch we’ve seen surveys created around the globe, and we’ve received lots of helpful feedback coming through about how to make Questions better suited for you.

Questions is still a baby right now, or an MVP if you prefer. We’d like to involve you in prescribing the roadmap so on the day of the launch, we sent out a questionnaire asking about how you currently use other online surveys and your thoughts about this research method. The results were interesting — so interesting, in fact, we thought we’d share them with you.

Our survey questions

The online survey we created was simple. It contained four questions — three open ended, and one multiple choice. We wanted to find out how people currently use online survey tools in their research, and what features would make a truly great survey tool for user research.

The questions we asked were:

1. If we could give you one awesome feature for an online survey tool that would make your life easier and your work more delightful, what would it be and why?
2. What is your favorite question to ask people in your research?
3. Can you tell us how you currently use online surveys in your research? Or, why you don’t?
4. If Optimal Workshop became a band, what would you call us?

• Click 182
• Red Hot Chilli Sorts
• Rage Against The Tree
• Radiobutton
• The Rolling Iterations

Only one question (the first one) was mandatory. We did a quick test with some colleagues, then hit the big green ‘Launch’ button.

Our participants

As we wanted to hear from the Optimal Workshop community directly, we thought what better way to talk to you all than an email announcement about our latest tool?

If you’re part of our email list, you’ve probably seen this before:

Since we recruited participants through email, we had a rather high abandonment rate. This can happen a lot with these types of recruitment methods — people just get curious and take a peek at the survey, then exit out as quickly as they opened up the link. Fortunately, we still collected pages and pages of data from those who participated.

152 people completed our online survey, and many answered in great detail!

The top countries where our participants were located were:

• United Kingdom
• United States
• New Zealand

Our results

There’s no set end date in mind for this survey, so it’s still live. In fact, at the time of this analysis, we still had answers rolling in (maybe a lot of you are still catching up on emails from August!).

Since the survey only has four questions, the median time taken was naturally only 3.5 minutes.

1) What do you think makes a great online survey tool?

In the first question we asked: If we could give you one awesome feature for an online survey tool that would make your life easier and your work more delightful, what would it be and why?

We received tons of different ideas ranging from cool data visualizations to gamification. We also saw a few similar ideas too. One of the most popular was adding photos to individual questions, similar to what we have available for OptimalSort’s cards. This would allow people to ask questions about a particular product, design, or portion of a site.

Another popular suggestion is to provide full survey templates that researchers can use for their own surveys and customize as needed. This will help with ideas, consistency, and if we get it right, with quality.

A lot of participants stated that survey logic is important, so this is in the works already. For those unfamiliar, enabling question logic allows participants to skip through questions depending on their previous answers. This helps to eliminate irrelevant questions, keeping people engaged with what they’re doing.

These were some of the many other features that you said would make a great online survey tool:

“Sentiment analysis on answers. This might give data analysts an idea on how people subconsciously react to a question rather than what they tell us.”

“Help people use surveys for the right thing. Help them ask great unbiased questions.”

“I would create some kind of automated algorithm to automatically code open questions.”

2) Your go-to research questions

The second question asked: What is your favorite question to ask people in your research?

One of the most popular questions you like to ask your own participants is “why” in a myriad of ways. In fact, there were 20 questions that contained the word “why”.

“Why are you visiting this website?”

“If could could change one thing about X, what would it be and why?”

“How do you feel about X and why?”

We also saw a few funny ones too.

“If you had a magic wand and could change anything you wanted… (And why)?”

And the ever-important:

“What is your favorite dog breed?”

3) How you use online surveys in your research

Our customer base is made up of people from all around the world in many different industries, so it’s not surprising to see that you all use online surveys in a multitude of ways.

When we asked you “Can you tell us how you currently use online surveys in your research? Or, why you don’t?” we received close to 80 responses.

After reviewing the data, we saw that you mostly use online surveys to quantify opinions, attitudes, and needs. For example, using online survey forms to gather information to better understand the needs of your users; or get opinions about a product or service.

Second to that, many of you use online surveys for participant screening to complement your other user research, and to better understand a particular demographic. Recently, we launched a new participant screening beta feature for our tools, which allows you to accept and reject participants based on their answer to your screening question. It’s great for targeting a particular demographic and getting the quality responses you’re looking for.

Overall, the results showed us you use online surveys to better understand who your users are, and what they think about whatever it is you’re researching.

Welcome to the stage…

Finally, we asked you what you’d name our band, if the Optimal Workshop crew decided to take to the stage and swap keyboards for keytars.

Looking at the results, it wasn’t even a close race — The Rolling Iterations won by a mile.

So, we’d like to welcome to the stage The Rolling Iterations!

Thanks again to everyone who participated in our Questions survey — we love hearing about how you use our tools, and how you do your research. If you’d like to help us improve our tools, join our research panel and voice your thoughts and ideas.