Tree testing in the design process — Part 2: The design phase

In Part 1 of this series, we looked at:

• Where Information Architecture (IA, in general) and tree testing (in particular) fit into the overall design process
• How to best combine tree testing with other methods in the research phase of your project

In Part 2, we’ll look at how tree testing fits into the design phase of a project: creating several alternative site trees, testing them against each other, and iterating until we get a high-performing site structure.

From research to design

Prime time for tree testing is early in the design phase, once we’ve done enough research to feel we have a good handle on our audiences, their background, and their needs.

We start creating drafts of new site structures immediately after we finish a content audit – that is, after we decide which content we will be adding, updating, or deleting. While content is always a moving target, it really helps to have most of it identified before trying to design a structure for it.

This work on content and structure can be done in parallel with conceptual design, but usually comes before more detailed work such as page layouts, fine-grained interactions, and visual design.

Creating new trees

From our research, we should have several ideas about what to change (and what not to) in a new site tree – not just grouping, but labeling too.

We can then start sketching out new trees by looking at these ideas and our list of planned content. While the creation of site trees is a whole other discussion, we can at least say that it’s typical to rough out two to five different trees at this stage, down to level 2 or level 3, just to explore how they might work. We might do this ourselves, or (even better) we might involve the whole team to get a wider variety of informed ideas.

In the design phase, we can increase the quality of our site tree by doing two critical things:

• Going wide (testing several alternative trees at the start)
• Going deep (testing and revising down to a single tree that performs well)

Let’s look at each of these tactics in more detail.

Going wide

In the design phase of a project, it’s a mistake to create just a single new site tree.

Testing several alternatives at once

It’s too early to finalize our thinking; we probably haven’t done enough research and we haven’t actually tested our single idea with users. If it doesn’t work out, what do we do – start over? If we only create (and test) a single site tree, we’re taking a big (and unnecessary) risk.

The smart thing to do here is “go wide” – that is, generate several different site trees to exercise our various ideas, then pick the two or three most promising trees to test against each other.

This may seem like a lot of extra work, but it’s not. It turns out that trying to settle on a single tree at this stage is usually very difficult. There are different groupings to try, different terms to try, and (if we’re working with a team) diverging ideas from other members. We’ve found it easier to generate several trees that incorporate these inputs than it is to try hacking and slashing them all in a single structure.

By “going wide”, we raise our chances of hitting on the best design. We start with several candidate trees, then use a reliable method (tree testing) to choose the best one.

For more on going wide in UX design, see Jared Spool’s short article on exploring multiple variations.

Data for the CEO

Consider also that the “other members” that we get input from might include the CEO. And (trust us), if we need to shoot down their out-of-left-field idea, it’s much easier to do that with objective data from testing than just our personal opinions.

Guarding against genius design

Perhaps most importantly, testing lots of ideas early on avoids the peril of genius design. By that, we mean designers who believe they are talented, and tend to believe that all their ideas are good ones.

If we have a “genius” designer on the project, we must tread very carefully. Because when a designer falls in love with a single idea early on, it’s really hard for them to get away from it.

Learning from other industries

Fixing on a single idea too early is not a new problem. Other industries encountered this (and solved it) years ago.

Consider traditional graphic design and advertising. In both professions, the classic approach is to go wide in the early conceptual phase:

• Brainstorm dozens of ideas
• Explore 5-10 in more detail
• Pitch the top three to the client

It brings to mind the advertising person who said that if they went away and worked and then came back with a single idea to pitch, they would be fired on the spot.

Going wide – an example

For an example of testing alternative ideas, let’s look at Meridian Energy, a renewable-power company that needed to redesign its site tree.

When they ran an open card sort with their users, the results suggested that the current top-level headings didn’t match their mental model:

So they brainstormed a bunch of ideas, and decided that two of them were worth testing. They set up three tree tests — the two ideas plus the current site.

When the results came back a week later, the current tree performed poorly (as they expected). But so did the two new ones:

Obviously, they were not happy about this. Time to start over, right?

Well, not really. When they dug deeper into the results, they found patterns in both trees that performed very well. The problem was that there were other patterns and potholes that sabotaged the overall scores.

So, in the second round, they created a third tree that was a hybrid of the best of the earlier designs. And the third tree proved to be the charm:

Going deep

Making several site trees compete against each other is great, but at some point, we need to reduce them down to a single high-performing tree.

Iterating until we get it right

Once we’ve run our first round of tree tests, on two or three of the most promising trees we thought up, we analyze the results. Typically we find:

• One of the trees performed the best, but it still has some problems to solve.
• Some of the trees just didn’t work for our participants. We can safely discard these trees and move on.
• Some of the lower-performing trees may have elements (certain groupings or terms) that actually did perform well, so it may make sense to incorporate these elements into our best tree.
• In rare cases, we may find that none of the initial trees perform well enough to continue with. We’ll have to come up with some new ideas quickly (perhaps ideas that were discarded earlier) or go back and do more basic research with users.

After the first round of tests, we either have:

• Two trees that performed reasonably well, but should perform better with the revisions we have in mind, or
• One tree that clearly outperformed the others, and should improve further with our revisions.

We can now run a second round of tests to see if our revisions did indeed make things better:

• If we tested two trees, we can then determine which performed better and which we’re more comfortable going with as our actual site tree.
• If we tested one tree, we can look for any remaining areas or terms that need tweaking.

In a perfect world, we would keep testing until we had a perfect tree, but there is never time or budget enough for that. We typically only do more than two rounds of testing if there are important parts of the tree that are still not performing well enough.

Keeping it cheap and fast

If we were expecting tree testing to be a one-off trick – build a tree, test it, and we’re done – it may be alarming that we recommend several rounds of testing, with several trees, winnowing and refining them until we get a single high-performing tree.

What makes this approach feasible is that we now have mature testing tools that are both:

• Cheap (affordable for any development team), and
• Fast (able to deliver results in a week or two)

The combination of cheap and fast changes how we should approach design. Instead of doing a single round of deluxe in-person testing (or worse, no testing at all), we can do several cheap online tests in less time and for less money.

There are cases where in-person testing is the way to go, particularly for complex interactions, or for when there are only a small number of participants available. But for most projects, several rounds of lightweight tests are a better bang for the buck.

And that means we can go wide at the start, and go deep through to the end.

Putting it all together

Let’s take a step back now and look at how we can fit card sorting and tree testing into our overall design process.

The recommended approach

If we have been given a reasonable amount of time to create a site tree, we do the following:

There are several advantages to this “full-fat” approach:

• Card sorting generates ideas for our new tree — ideas that come from actual users, not just the project team.
• Baseline tree testing shows us what needs fixing (and what doesn’t), and gives us a score to measure against later.
• Two rounds of testing for the new trees lets us experiment, refine, and raise our chances of creating an optimal design.

The hurry-up approach

All of us have worked on a project where we were not given that “reasonable” amount of time to do what was needed. Sometimes we have to cut corners.

Because we loath the idea of creating a site tree without some form of validation, we still run tree tests, but we abbreviate the process, using a single round that combines before and after:

Summary

Here are a few key points that I hope you take away from this pair of articles:

While information architecture spans the entire design process, tree testing is typically done early.

• In the initial research phase, we should employ contextual inquiry, an open card sort, and a baseline tree test of the existing site to give us clues to what needs changing in our site tree.
• In the design phase, we should “go wide” by creating several alternative site trees incorporating the ideas from our research, and make them compete against each other.
• Online testing tools are cheap and fast, allowing us to “go deep” (run several quick rounds of tests), winnowing and refining our ideas down to a single high-performing structure.