How we used card sorting to design a Style Guide for web developers and UX designers

Wynyard Group is a market leader in risk management and crime fighting software used in investigations and intelligence operations by government agencies and financial crime organisations. Wynyard Group has recently joined the journey to incorporate a User Experience (UX) team into their family.

Why a style guide? Why a card sort?

One of our first steps towards UX integration was to create a style guide that our web developers and UX designers can look to for styling, components and patterns. Our purpose was twofold: to help them create high-quality products that are visually and behaviourally consistent. And to free them up to focus on workflow, information architecture, and interaction design elements of the whole rather than the styling of individual parts.

Wynyard currently uses Confluence as an internal wiki for information, so we added a section to it called ‘User Experience’ which included the subsections ‘User Experience Tools’ and ‘Style Guide’. It then occurred to us that how we group and arrange elements of our style guide might be (probably will be) completely different to our web developers. Thus, we decided to run a card sort to make sure our style guide meets the needs of the audience. And because we’re always looking for a chance to test out new technologies, our journey into card sorting with OptimalSort began.

Getting started on the card sort and selecting participants

A great idea when starting up new testing projects is to go back to the basics. I’d heard of card sorts before but had never been directly involved in one. So I hit the books, and in particular, Donna Spencer’s book Card Sorting: Designing Usable Categories. After reading through this and researching a few tools we came across OptimalSort. Our developers were spread across Christchurch and Auckland, so having an online tool was definitely a requirement. Having tested it out, I found it was very quick, easy, and customisable. I was sold.

To pick our card sort participants, I went to our internal message board (Yammer) and looked at the members of our Engineering Guild — Web Guild. We had 50 members at the time, but this included a mix of marketers, UX designers, architects, front and back-end developers, and anyone else who was interested in the messages being posted up for this group. Of this I took a subset of 20 that were most likely to be involved in implementing our designs. So I recruited the people that would be taking our wireframes or prototypes and integrating them into current products or new products.

Creating and running a draft card sort

I kicked the process off by creating a card sort that I could test on colleagues to get feedback before I opened it up to our main participants. Some of the cards tested well, while others were a little confusing, and feedback was given. The bonus about this was that while they were completing the test online, I was able to stand in the room and watch, asking and answering questions around the cards.

As with most things you try for the first time, my sort wasn’t ready. One point that came out quite quickly was that I had combined some cards that were process, such as Information Architecture and User Research, and others that could be explored through workplace education (style guide importance). Therefore, I could remove these as they clouded the areas that I wanted participants to group around.

If at first you don’t succeed, eat a cookie and try again

I made changes to the cards based on the feedback I received, and decided to go with a very simple approach with a limited amount of cards. This was because our participants hadn’t completed a card sort before, the card concepts may have been relatively new, and I wanted to see if we got any convergence to start off with. It was also a double check to see if I had created the correct cards.

So my first official card sort looked like this:

What we discovered from the first open card sort

I published the sort, and emailed the link with an explanation out to our participants. And the results were … not what we had expected. To come up with this sort, I had ideated around base groups such as visual design, patterns, components and layout, then created cards to go under those categories. I was expecting very similar groupings and even category names to what I had come up with, but this was not quite the case.

OptimalSort has some really good analysis tools that let you get into more detail behind how the participants grouped the cards. The two tools that we focused on were Participant-Centric Analysis (PCA), and the Similarity Matrix.

This is the PCA, which displays common grouping among all participants, and some of the different labels.

And here’s the Similarity Matrix:

Overall we had 16 responses, with 4 abandoned. We ended up including 2 of the abandoned results as they were fully complete but were not submitted. So all together that made for 12 participants from our web development team.

From these we re-grouped and discussed the results. The first word to jump out was ‘Prettification’. Although this was the main grouping across participants, we decided to use ‘Look & Feel’ as we felt it connected more with our goals. We also didn’t want to associate visual design with the limitations of prettification, as it is much more than that.

It was interesting to see that the cards tended to be grouped by overarching concepts of what the cards were used for (such as ‘Navigation’), although more specific concepts such as ‘Components’ were used. The groupings were a cross between what we would have done in User Experience, and what the developers would call things.

Then we ran a closed card sort with new categories

So then we decided to run a closed card sort.  We decided to add more cards to see if there was convergence towards the categories they had made, and whether people could group what we believed were easier (ie. Buttons) versus the more difficult (ie Search) cards.

Most of the categories were taken from the results of the previous card sort, but patterns for us were also a very important concept that we wanted to include. By including definitions, we wanted to see if these group concepts were understandable:

• Components — Ready-made common UI Components for input and functions
• Look and Feel — Creating a consistent looking User Interface
• Patterns — Standard, pre-packaged design solutions to common workflow problems
• Navigation — Moving between Apps, Screens and within Pages
• Structure — How to set up and lay out an Application or Page
• I do not know — None of the provided categories seem right

Some of these things are not like the other things, some of these things just don’t belong…

The closed sort ended up with 10 completed responses and 4 abandoned. Below is a Popular Placement Matrix. It let us see very clearly where cards had been grouped, and the level of uncertainty around each.

Our participants were relatively clear around what could be grouped under ‘Components’, such as Checkboxes (100%), and Buttons (100%). They also had high placement confidence around ‘Look & Feel’, such as Colour (100%), Icons and Typography (90%). The more complicated concept of Responsive Design, which we viewed as a more difficult card to sort, had a fairly even split leaning towards patterns.

Some interesting points to note include that 40% thought that Search was part of ‘Navigation’, or ‘Components’, and only 20% thought it was a ‘Pattern’ (which is where we placed it). The link to navigation could be because people associate search with navigation when they can’t find what they are looking for, or the information architecture has failed. It was also good to note that a majority of the cards were sorted into groups, instead of going into the ‘I don’t know’ category.

Below is a Results Matrix which clearly shows how often cards were sorted into each category.

The Results Matrix also shows high confidence around the placement of components and visual elements. This gave us relatively good confidence that if we had an area called ‘Visual Design – Look & Feel’, and ‘Components’, our web developers would know what they might find within it.

But we also had to acknowledge the uncertainty around some of the cards, shown by the low scores across a range of groups. We decided that ‘Structure’ was too complicated a concept, as some of the things that had been put under there were patterns, and there didn’t seem to be high confidence around placing cards in this category (other than forms). ‘Patterns’ was also not well understood, which validated the need to have workplace education and advocacy around them to raise awareness, as they were a grouping we wanted to keep.

Overall we had some strong groupings, and some that would need changing or updating, and some that would involve further research and learning.

Next steps

Overall the card sort was a great learning experience because it cemented the fact that our UX designers and our web developers have a crossover of terminology, but we also have differences. To get the best of both worlds, and to be able to present consistent groupings, we will have a mix from both, where some will require more description and learning than others.

Next steps when we pick up the style guide again will be to present what we have done internally to increase understanding. Then, depending on our direction, we’ll run a tree test using Treejack to find out how our style guide structure is working, and if the same people can easily find what they are looking for. Tweak, test, rinse and repeat.

To sign off, here’s me in action: