Is closed card sorting an outdated technique for IA?

In this post, our guest blogger David Juhlin, user experience consultant at The User Experience Center, discusses his own observations with closed card sorting, tree testing, and first-click testing in relation to improving information architecture (IA). David compares each method to closed card sorting and shares his opinion as to why closed card sorting is becoming outdated.

There are many different techniques to evaluate information architecture, and some of them overlap regarding the insights they provide. The closed card sort is one such method that has a large overlap with others. While the closed card sort provides a lot of value, much of it can be gained from other methods that also provide additional insights. Let’s take a closer look to see if the other methods are enough to deem the closed card sort an outdated technique for evaluating IA.

For those unfamiliar with this methodology, closed card sorting is when participants are provided with cards that either have something written on them or have pictures of items. For example, if we’re interested in how a grocery store is organized, we would provide the participant with cards such as “cucumber”, “milk”, etc. You then ask the participant to sort these cards into predetermined categories you have created. The categories in this example would be “Produce”, “Fridge Items”, “Frozen Items”, etc. If most participants place “cucumber” under “Produce”, you can be fairly sure this is a good category with respect to finding the cucumber.

Open card sort versus closed card sort

In a closed card sort, participants are provided with the categories into which they sort the cards. However, in an open card sort, there are no predetermined categories and participants are given cards and asked to sort them in a way that makes sense to them.

Once the participants are done sorting in an open card sort, they usually label each group. For example, one participant might sort “cucumber”, “tomatoes”, and “lettuce” into one group and label it “salad ingredients”. Another group might create a label named “condiments” containing “ketchup”, “mayo”, and “olives” (sometimes you don’t always agree with how the participants sort items, but they are allowed to sort it however they see fit). The point of the open card sort is to gain an understanding about how participants group certain items.

While appearing similar, both methods do have many differences. The fundamental value in each method is different. The open card sort tries to capture the mental model of your participants, i.e., how they structure the information. This is a form of exploratory study.

The closed card sort, on the other hand, is more of a type of validation research. Therefore, you would use this method when you have an idea of how the users will structure the information. At this point some will disagree, since it can be used for exploration too. I don’t disagree — you can use the closed card sort for exploration, but the research methodology is based on validation. So, if you have multiple design structures competing with one another, you are still validating and investigating which one is performing best. Even if you add moderation to your closed card sort to gain an understanding of how the participant thinks, the basic method is still a validation since you see how participants react to your intended structure.

When comparing closed and open card sort, the closed sort is still a relevant method since both options are so different.

Tree testing versus closed card sort

Tree testing is a method that provides the participant with an opportunity to test multiple levels of the IA. The purpose behind tree testing is to see how participants navigate your information structure to find answers to a specific task.

An easy way to think about it is if you asked a participant to navigate all the folders on your computer. You have many subfolders nested within each folder — similar to an information architecture structure on a website. Then, you ask your participant to find your photos from your vacation in Greece. As participants click around in your folders, you collect information about how they navigate your folder structure. The tree test does the same, but it just uses your intended information architecture instead and presents it as nested elements. The point of the tree test is to see where participants expect to find certain information in your information architecture structure.

Both tree testing and closed card sorting are validation methods (in contrast to the open card sort that is an exploratory method), so these are best used when you have an idea of how you think users will structure the information. Although mechanically different, the methods are still very similar. With the tree test, you have to build a tree and the first level nodes/branches would be the exact same as the categories you would use in the closed card sort.

As an example, say you conducted a tree test of a grocery store information architecture and the participant was asked to indicate where he or she would find a “cucumber”. Most participants would start out in the “Produce” section of the IA tree. If you conducted a closed card sort, most participants would place the “cucumber” card under the “Produce” category. As you can see in this example, both methods would provide the same insight that participants think the cucumber should be found within the produce section of your information structure. Therefore, both tree testing and closed card sort provide you with the same information about the top-level information structure.

With tree testing, you actually gain more insight because this methodology allows you to analyze paths further down in the IA structure. At this point, it may seem as if the closed card sort methodology is not worth using anymore because the tree test provides the same information — but not so fast! Yes, you get all the insights from the tree test, but it takes participants a longer time to complete the same task, since it requires them to look further down the tree structure. This means if a participant is performing 10 tasks on the tree test, the same participant could have sorted 30-40 cards in the same time span. If the participant uses the same time span for one or the other methodologies, the closed card sort has been able to cover more items in the information architecture, and therefore it provides a broader insight about the top-level structure, thus providing additional value.

In addition, if you’re conducting a tree test, you need to have an information structure to test (the entire folder system). If you have an old structure then you’re off to a good start, but if you need to create a structure from scratch there will be additional work associated with this methodology compared to the closed card sort that only needs a top-level structure. Therefore, the closed card sort can be quicker and allows for faster iterations. However, the benefit of building the structure and taking the extra time is that you are forced to think it through. This might make you realize there is a need for an additional top-level category

As you can see, even if there are overlaps in insights gained between these two methods, there are unique benefits to both. While the tree test method provides a narrow insight of the top structure as well as the deeper levels in the information architecture structure, the closed card sort provides broader feedback of the top-level structure. However, a closed card sort does not provide insight about the levels further down in the information structure. In addition, the closed card sort is quicker, especially if you need to build out the information structure of the tree.

First-click testing versus closed card sort

As some of you might know, a first-click test involves uploading an image — a screen shot, sketch or anything in between — and asking the participants where they would click first in order to perform a task. In my grocery store example, if we presented the participants with a screenshot of an online grocery store and asked them to find a “cucumber”, most participants would likely make their first click on a menu heading labeled “Produce”.

The first-click test is, in the same way as the tree test and the closed card sort, a validation method with the purpose of testing how well the intended design works for the user. As with the tree test, the first-click test actually provides a different insight, since you have a screenshot and can also test the layout/position of different items. Additionally, the mechanics of a first-click test are quite simple for the participants, so each task is performed quickly.

The efficiency of a first-click test is about the same as a closed card sort, since both test can be finished pretty quickly.

The only drawback of the first-click test, in comparison to the closed card sort, is the need to have some type of design. However, this can be an early wireframe or sketch — it doesn’t have to take much time to create.

As a result, both methods provide about the same insight for the top-level IA, but the first-click test provides deeper user insights since it may reveal if some design elements of the page might be distracting. Both methods can also be very quick to set up, if you don’t spend a lot of time designing the screenshot. Therefore, the first-click method could replace the closed card sort. The only time I would use a closed card sort is if I don’t have any idea about how I would like to structure the design visually or really just want to isolate the top-level structure from all other influences.

Is closed card sorting outdated?

In many cases, the findings from tree testing and first-click testing will overlap with the insight gained from a closed card sort, and often provide additional value. So I conclude that if a project requires validating a given information architecture, my preference will be to conduct a tree test or a first-click test rather than a closed card sort.

Of course there are many other productive uses for closed card sorting, so while the method seems to have been superseded in the IA context, it has plenty of life in it yet!