Tree testing and card sorting Yelp’s support content

Now, when we view the pietree, we can instantly see a positive pattern emerging. The green paths represent the correct direction, and the yellow represents the correct answer. As you’ll see below, the green path is thick and dominant, which tells us quickly that users completed the task successfully.

We can also see that even though some people went down the wrong path, most went back up the tree (indicated by blue) because they recognized they’d gone the wrong way. The pietree enables us to see within seconds the success of the task. The yellow circle is large, which represents the 94% of participants that found the correct answer.

Two key findings from the results of the two lowest scoring tasks

Two of the lowest-scoring tasks gave us truly useful insights that could influence our redesign.

Participants went to multiple places to complete the same task

In Task 1, participants needed to find out what to do if when they saw a bad review of their business. This task received an overall score of one, and took participants an average of 59 seconds to complete — longer than people usually spend when looking for information on a website.

In fact, only 11% landed on the correct destination, and only 22% went down a particular path without backtracking. So 89% of participants went to the wrong information, and 78% of participants went the wrong way at least once. Looking at this chart gives us the information instantly.

One quick look at the pietree tells the same story. Participants went all over the place looking for the correct destination, and the very thin green path shows us the small number that actually got there. Another key warning sign for us is the amount of red in the central ‘Support Centre’ landing page. This means that participants often got both their first clicks and subsequent clicks wrong as they moved back through the tree.

And when when hover over the centre of the pietree, we’re able to see in detail how often people visited the ‘Support Centre’ homepage. Even though only 35 people completed this task, the homepage was clicked on 131 times, which shows how often participants felt the need to go all the way back and start again.

The third-level headings are ambiguous and causing confusion

In another task, participants needed to locate their Business ID within their accounts. This task received an overall score of one: only 17% of participants selected the correct destination, and only 22% of participants clicked through a path without backtracking. The task took people an average of 41.2 seconds, which is again a very long time — especially when most participants didn’t get to the correct destination.

And then when we look at the pietree, we’re able to see something particularly interesting: there are two large circles in the centre instead of just one. This shows us that the second level label,‘Yelp for Business Owners’ was the most popular first-click from the homepage ‘Support Centre’ label.

It’s also the correct path, and so the size of the circle would be a good thing…if it wasn’t for the huge amount of red! The red indicates that most people went the wrong way after this point.

When we hover the mouse over the ‘Yelp for Business Owners’ circle, we’re able to see the data in more detail. We can see that of the 125 times participants clicked on ‘Yelp for Business Owners’, 107 of these clicks took participants down the wrong path:

So to find their Business ID, participants clicked on the labels ‘Account information’, ‘Claiming your business page’, ‘Business owner features’, and ‘Business owner FAQ’ far more often than they clicked on the correct location, which was ‘Updating your business page’. This is compelling evidence that shows the ambiguity of labels presented to participants once they’ve clicked on ‘Yelp for Business Owners’, and will definitely be useful data for our redesign.

Creating and running an Open Card sort on Yelp’s support site

Tree tests are great for establishing how effective an information architecture is based on user tasks. Open card sorting is used for a totally different purpose, and can’t be used to prove or benchmark an existing IA. Open card sorting is great for getting ideas for organizing and categorizing the content you have to include.

So after running the tree test, we ran an open card sort on a selection of the support site content to find out how people would naturally group and label it. We selected 40 pages out of the 295 total pages that are aimed at people whose businesses get reviewed on Yelp. And we told participants to consider themselves as business owners in the instructions.

We narrowed the number of cards because expecting participants to sort 295 cards would almost certainly result in a huge abandonment rate. And because the data we’d get from a sort that big would likely be unweildy and difficult to draw conclusions from. A possible next step (which we didn’t take) would be to run separate card sorts on the rest of the content to add to the insights.

For this study, we wanted to:

• find out if most participants agreed on any particular category or label
• get ideas for what labels and hierachy to use in the new information architecture.

57 people completed the card sort, and it took them an average of 8.3 minutes.

Standardizing the categories enabled us to to see common themes

In OptimalSort, you can standardize the categories to make sure all the commonly named categories are counted together in the data. For example, we created a standardized category called ‘Yelp deals and vouchers’, based on the label suggestions you can see on the right. We’re then able to see that some of these cards were placed under that label up to 10 times.

When we look at the actual support site, we can see that Yelp categorizes a lot of this content together. Users click twice to get to this view. There’s also some content on here that might be better placed elsewhere, and the amount of links might require further hierachy or structure.

Standardizing another set of categories shows us something different. 18 participants placed the following cards into categories that can mostly be standardized as ‘Business information’ or Business FAQs’. These labels are reminiscent of the overused and totally ambiguous label ‘General information’ that should be avoided at all costs by information designers. A useful question for us to confront is why these particular labels don’t lend themselves to more particular categories for these participants.

Almost 50% of participants made similar categories to each other

The PCA (participant-centric analysis) shows the top three most representative IA submissions by participants, and as tested against all the other categories. Below is the PCA that had the most agreement out of the three — 21 out of 44 participants created an IA similar to this one.

This IA has four categories. These four categories might be a good place to start when redesigning the IA of Yelp’s support page.

This dendrogram uses the Actual Agreement method to show that X% of participants agree with a particular grouping.

If we zoom in on the bottom left corner, we can see that 70% of participants agree that the three highlighted cards should be grouped together under a heading that contains the word ‘Deals’:

And if we cast our eyes a little higher, we can see that 54% of participants think that these three highlighted cards should be grouped together under a heading that contains the words “Bad Reviews’:

These results are some of the patterns that jump out, and that we could take into account in our redesign. The way results are displayed makes it quick to get insights like these.

So, what comes next?

Yes, I’m leaving you hanging — I warned you at the start there’d be little in the way of resolution. The next steps involve using the data to influence the decisions we make in the redesign. And after we’ve created a new potential IA that we feel addresses all the problems the results have shown us, we can run another tree test with the same tasks, and compare the results.

If you’d like to look at the actual results, and either suggest your own design recommendations and offer further (or different) interpretations of the data, email kathryn@optimalworkshop.com.

I’ll give you access to the actual results, and potentially publish your insights.