Comparing Open and Hybrid Card Sorts with Tripadvisor.com

Millions of young people across the US and Canada are gearing up for the fun-filled week called Spring Break. Traditionally coeds flock to sun-drenched beach and tropical destinations, like Daytona Beach or Panama City, but others might be looking for slightly less overpopulated and more exotic locales, such as Egypt or Bali.

Major travel sites like Expedia or Travelocity are great for scoring great deals on airline tickets and hotel rooms, but for those looking for inspiration for their next Spring Break getaway, there is a hidden gem of a site buried within the top mega menu on Tripadvisor.com called “Trip Ideas”.

Once you find it, you are greeted with a landing page asking you What You Like. The page presents the reader with somewhat arbitrary categories, ranging from geographical characteristics (Beaches & Sun) to things to do (Shopping, Skiing), and even a category for “Family Fun”.

Depending on how big your screen is, you might miss out on the second question Where You Want To Go, which requests that you choose a general location on the planet (or Anywhere). Once you make a selection, you will be rewarded with a sortable list of popular vacation spots matching your specifications. Each location provides you with a wealth of information important to a potential visitor, from maps to weather to breathtaking photography.

Earlier this month, Optimal Workshop introduced the “hybrid” option for card sort surveys, which allows researchers to specify categories ahead of time for survey participants to use, while still allowing them to create and name their own categories. This is different from both an “open” card sort (which requires participants to create and name all their own categories) and a “closed” card sort (which only allows participants to sort cards into categories you give them).

• Open card sorting: Participants can create and name their own categories
• Closed card sorting: Predetermine your categories
• Hybrid card sorting: Predetermine some categories but also allow participants create their own

To see if visitors to the “Travel Inspiration” area of Tripadvisor would understand how destinations were categorized, I selected the top 4 to 6 destinations in each of the nine “Travel Inspiration” sections and set up two card sort surveys, identical except for making one an open sort and the other a “hybrid” sort. This kind of testing is commonly called a split testing or A/B testing, which keeps all variables the same except for one that is being analyzed. In this study, I will look at the differences between the two study formats (if any), as well as the overall implications for Tripadvisor’s categorization setup. It is important to remember that this is not a comprehensive study of Tripadvisor’s website, but only looks at categorization of a small area of content. When undertaking user research, I always try to corroborate results using multiple methodologies.

Results

Each study had 36 cards to sort. We recruited participants through Amazon’s Mechanical Turk and received 50 completed sorts for the open sort and 53 completed sorts for the Hybrid study. When I set about analyzing the results I was immediately stuck by how fast participants in each study completed the assignment:

Open sort:

Hybrid sort:

Card sorting is all about gut reaction and instinct, and that was clearly evident in how participants sorted the open sort cards. After standardizing the list of categories generated down to 175, it became clear that there were several different kinds of organizational models used by the responders to complete the sort:

By place

Of the top 10 categories displaying the highest amount of agreement, 6 of them referenced specific places, such as Canada (1.00 agreement, 2 unique cards), USA (0.84 agreement, 15 unique cards), and Americas (North & South) (0.79 agreement, 19 unique cards).

By physical description

Another popular method of sorting was by some kind of physical quality, like Islands (0.78 agreement, 6 unique cards), City (0.79 agreement, 19 unique cards), or Beaches (0.54 agreement, 17 unique cards).

By the picture on the card

A few categories had a high agreement value for groupings that seemed rather arbitrary. This led me to believe that some of the participants sorted in a way that was at least partially influenced by the picture I used on each card. This is a valuable insight, and indicates that if you use a photo on each card, be aware of the potential impact it may have on your results.

By generalized category

Participants were instructed to sort the cards, keeping in mind that this was content for a travel and vacation website. There were many categories with generalized names like Far East (1.00 agreement, 3 unique cards), Tropical Destinations (0.82 agreement, 14 unique cards), or Exotic Locations (0.43 agreement, 30 unique cards).

Table 1. Four different models of sorting cards

Open & Hybrid sorting methods compared

Turning my attention to the Hybrid sort, I saw right away that the vast majority of participants chose to use the categories I already provided for them, with the exception of a few outliers (mostly garbage text like tghgh or leftovers of the default text Click to rename).

Unfortunately the categories already being used on the Tripadvisor website didn’t seem to resonate well with the participants, with none of them achieving more than 0.45 agreement (which was Beaches & Sun). The categories that described a physical characteristic of the location seemed to fare better, but on the whole the responses showed very little consensus on what should belong in the Adventure versus Family Fun categories (in the case of Grand Canyon, Arizona), or whether a destination like Lake Tahoe, California should be in the Beaches & Sun or Spa categories.

So, given what I’ve already discovered between these two sorts, what should Tripadvisor do? Before making a recommendation, we should consider the Similarity Matrices for each study:

The Similarity Matrix is a fantastic way of getting an overall impression of clusters of similar cards. I always look first for areas of darkness, which indicates high agreement between pairs of cards. In both studies, the beach and tropical oriented destinations were consistently grouped together, and this finding is supported by categories with high agreement in both sorts. Both studies also tended to group together European destinations, which is also supported by the categories used or generated by the participants.

We can also see some localized agreement within cards from the same geographical location, such as destinations in Mexico or Canada. Again, this echoes the different types of models used in the open sort.

The strong correlation between Atlantic City, New Jersey and Las Vegas, Nevada jumped out at me, especially since it appeared in both sorts. Curious, I went back and checked to see what categories the participants put this pair into. In the open sort, the categories were all over the place, but
North America appeared the most out of all of them. In the Hybrid sort, participants overwhelmingly sorted this pair in to the
Casinos category.

Summary

The contrast between these two studies really illustrates the principle of recognition vs. recall. If we take the pair of casinos as an example, we see that participants struggled to agree on a category name in the open sort. When provided with a short list of options, the category Casinos stood out as an attractive placement.

Another example is this cluster of Canadian destinations, which were grouped together fairly consistently in the open sort, but with a wide variety of names that nobody really agreed on (Canada, Mountains, even Land of Maple Syrup). In the Hybrid sort, these locations appeared almost exclusively in the Skiing category.

Overall, Tripadvisor’s method of organizing and presenting potential destinations is adequate but could use some enhancements.

The wide variety of attributes that participants seemed to assign to locations, especially place- or physical description based ones, seem to be the ideal foundation for a faceted search option.

Tripadvisor’s left navigation stack appears to sort destinations by generalized category and then by geographical location, but it doesn’t allow visitors to find all destinations by location (such as everything in the Caribbean), regardless of generalized category (I can only retrieve all Beaches & Sun destinations in the Caribbean, for example).

Tripadvisor does recognize that destinations can belong in more than one category, but loses out on the value of cross-linking between categories. For example, Aurba recommended for the Casinos, Family Fun, and Romance categories. However, instead of getting a list of all other locations that are also in the Casinos category, clicking the link will get you a small modal window with some content describing what makes Aurba a good Casinos destination instead.

Tripadvisor – help the spring breakers out by adding a few more hyperlinks and some faceted search functionality. I’d love to hear below if you have other recommendations for Tripadvisor or have any questions about the research.

Thanks,

Gianna LaPin

---

Gianna is the senior UI/UX designer for a leading medical institution’s intranet team. She leads large design projects for clinical and operational departments, and helps author and implement enterprise-wide standards for web-based communication. She also designs and conducts user research studies and evangelizes for the human side of human-computer interaction.

Want more website reviews? Check out our Mayo Clinic and WHO review.