How to benchmark your information architecture

As an information architect, I’ve worked on loads of website redesigns. Interestingly, every single one of these projects has come about because the website has had navigation problems. But before you go ahead and change up your information architecture (IA), how do you figure out whether the new navigation is any better than the existing one? How do you know if it’s worth the hours of design to implement it?

In this article, I’ll walk you through how to benchmark a site navigation using tree testing.

The initial groundwork

When you start any project, you need to identify success metrics. How would the project sponsor (or your client if you’re in an agency) consider the project to be a success? What KPIs determine how the project is doing?

Put your stake in the ground, draw a line in the sand — or whatever metaphor you wish to use. This is how you objectively determine how far you’ve gone from where you started. At the same time, benchmarking is the perfect exercise to figure out the areas that need improvement. To do this, you’ll need to lay down the groundwork.

If you’re benchmarking your IA as part of a web redesign project, great! Hopefully, that means you’ve already gone through the exercise of determining who your users are and what it is that your users would be doing on the site. If not, it’s time to find out. User research is a crucial part of benchmarking. If you don’t know who your users are and why they’re on your site, how can you improve it?

Of course, everyone has a different approach to benchmarking information architecture. Different navigation problems merit different solutions. This is one that I’ve talked myself into for a global navigation project and it’s worked out for me. If you have a different approach, please share! I’m always open to new processes.

Without further preamble, here’s the quick rundown of my approach to assessing and benchmarking a site navigation:

1. Conduct user research with the end goal to identify target users and user intent
2. From user research, determine at most 8-10 primary user tasks to test with the identified target users
3. Tree test the existing navigation with target users – using those user tasks
4. Tree test a competitor navigation with target users – using the same user tasks
5. Tree test a proposed navigation – using the same user tasks.

Step 1: Know who your users are

If it’s a new project, who is your target audience? Set up some kind of intercept or survey to find out who your users are. Find out what kind of people are coming to your site. Are they shopping for new cars or used cars? Are they patients or healthcare providers? If patients, then what kind of patients? Chronic care, acute care? If the project timeline doesn’t allow for this, discuss this with your project stakeholders. Ideally, they should have some idea of who their target audience is and you should at least be able to create proto user segments to direct your discovery.

If you have more than one user group, it’s best to identify the primary user group to focus efforts. If you have to design for everyone, you end up getting nowhere, satisfying no one. And guess what? This is not a novel idea.

“When you design for everyone, you design for no one.” — @aarron http://t.co/JIJ2c82d

— Ethan Marcotte (@beep) May 24, 2012

Your project stakeholder won’t like to hear this, but I would start with one user group and then iterate with additional user groups. Focus your efforts with one user group and do a good job. Then rinse and repeat with the remaining user groups. Your job is not done.

Determine what your users do

Interview or survey a couple of people who use your website and find out what they are doing on your site. What are they trying to do? Are they trying to find out information about services you provide? Are they trying to purchase things from your online store? How did they get there? Why did they choose your site over another website?

Identify priority user tasks

From your user interviews, could you identify 8-10 priority user tasks that you could use? For this, we’re trying to figure out what tasks to use in a navigation test. What are the main reasons why users would be on your site? How would the navigation best serve them? If your navigation says nothing about your users’ tasks, then you have your work cut out for you.

Step 2: Tree test your existing navigation

How would you benchmark without some metrics? There are a couple kinds of metrics that we could collect: quantitative and qualitative. For quantitative, I’m assuming that you have some kind of analytics running on your site as well as event tracking. Track which navigation links are getting the most interaction. Be sure to use event tracking on both primary, utility, and footer links. Name them accordingly. Try and determine which links get the most interaction, on which pages, and follow where the users tend to end up.

Of course, with quantitative data, you don’t have a really good understanding of the reasons behind user behavior. You can make assumptions, but those won’t get you very far. To get this kind of knowledge, you’ll need some qualitative data in the form of tree testing, also known as navigation testing.

I’ve only used Optimal Workshop’s Treejack tool for tree testing, so I can’t speak to the process with other services (I imagine that it would be similar). Here are the general steps below — you can find a more detailed process in this Tree Testing 101 guide.

Create/import a sitemap for your existing site navigation.

For my recent project, I focused benchmarking on the primary navigation. Don’t combine different types of navigation testing in one — you can do that in a usability test. Here, we’ll just be testing the primary navigation. Search and utility links are secondary, so save those for another time.

Set up user tasks and questions.

Take the user tasks you’ve identified earlier and enter them into a tree test. From this point on, go with best practices when setting up your tree test.

• Limit to 8-10 tasks so that you don’t overwhelm your participants. Aim to keep your tree test to 15 minutes long or less so your participants don’t get exhausted, either.
• Prepare pre-study questions — These are a good way to gather data about your participants, reconfirming their priority and validating any assumptions you have about this user group.
• Prepare post-task questions — Use confidence and free-form feedback questions to feel out how confident the user is in completing each task.

For more tips on setting up your tree test, check out this Knowledge Base article.

Run your tree test!

• Do a dry run with someone who is not on your team so you can see if it makes sense.
• Do a moderated version with a test participant using screen-sharing. The test participant could think aloud and that could give you more insight to the findability of the task. Keep in mind that moderated sessions tend to run longer than unmoderated sessions so your metrics will be different.
• Execute, implement, and run!

Analyze your tree test results

Once you’ve finished testing, it’s time to look for patterns. Set up the baseline metrics using success rate, time spent, patterns in the pietrees — this is the fun stuff!

Focus on the tasks that did not fare as well, particularly the ones that had an overall score of 7 or below. This is an easy indicator that you should pay more attention to the labeling or even the choice you indicated as the correct answer.

What’s next?

From here, you can set up the same tree test using a competitor’s site tree and the same user tasks. This is helpful to test whether a competitor’s navigation structure is an improvement over your existing one. It also helps with discussions where a stakeholder is particularly married to a certain navigation scheme and you’re asked to answer: which one is better? Having the results from this test helps you answer the question objectively.

• here are the reasons why a user is on your site
• here is what they’re trying to do
• here is what happens when they try to find this on your site
• here is what happens when they try to find the same thing on your competitor’s

When you have a proposed sitemap, test it again with the same tasks and you can use these to figure out whether the changes you made changed anything. You can also conduct this test over time.

A few more things to note

You could daisy-chain one tree test after another to test an existing nav and a competitor’s. Just keep in mind that you may need to limit the number of user tasks per tree test so that you don’t overwhelm the participant.