100s of resources and a brand new IA: How open card sorting saved the day

At Bloomboard, we’ve been working on a project to improve our online resources for the K-12 teachers and administrators who use our platform. We already have in place a comprehensive ‘Educator Resources’ site aimed at teachers, but we’d like to add content for administrators (the principals and managers who run the schools).

I recently ran two remote studies for this project, and I’ve written both up as short research stories (for your interest and joy!).

• Part 1 describes the tree test I ran to assess the findability of content on the existing ‘Educator Resources’ site
• Part 2 describes the open card sort I ran to generate ideas for the structure and content of the new ‘Administrator Resources’ section of the site.

What follows is Part 2 on open card sorting.

Why we decided to run an open card sort

We had received some requests, both from content providers and schools, for us to share content that would help administrators be better administrators. Up to this point, we’d focused our taxonomy and navigation on helping teachers with professional development, and we knew that a shift in audience would require a new approach.

We knew we couldn’t just dump the resources in the marketplace with no tagging at all — people wouldn’t be able to find them unless they searched (and would therefore need to know what content was there!). We needed a new taxonomy.  I knew where we should start in creating this new taxonomy: with an open card sort.

I’d already run an open card sort over the summer to come up with a draft, but my reflections on how I’d set the study up had me wanting to iterate and run it again.

Here are the questions we set out to answer:

1. How different are resources an administrator is looking for from those that teachers look for?
2. How can we create an taxonomy for administrators that supports this different mindset?

Creating the cards and recruiting participants

Our cards would represent the resources we had available, which I’d already collected from providers waiting in the wings, and reputable providers I knew in the field.  I also emailed an old colleague who is now a vice principle, and he (delightfully) sent me a picture of his bookshelf.

Before I started the card sort, I already had a half-answer to my first question. Principals I’d spoken with said their professional development often focused on effective ways to coach teachers. Therefore, as well as needing resources for their administrative tasks, they also wanted access to the same resources teachers accessed for their own learning.

In the open card sort I’d run over the summer,  I’d included both the title and description of the resource on the card. In hindsight, I can see that participants found the cards overwhelming, and that most titles were descriptive enough to stand on their own.

So I kept the new cards simple: just the title, and, if I thought it was necessary for clarity, the subtitle. Once I had all the cards Samantha (my awesome mentor) and I took a look at all of them together. She noticed that many cards had the word ‘leadership’ on them. She said that this might create an artificial grouping because people will see that word and jump to a conclusion — whether they realize they are doing it or not!

With this advice, I monkeyed with a few of the titles a bit to prevent this bias. But, let me tell you, it’s really tough to find a word that conveys the same exact meaning as leadership!

Just like with the tree test, it was nearing holiday time and I knew administrators are often busier than teachers. I was concerned I wouldn’t get enough participants, even though I only needed 20. I emailed 100 educators to start, but didn’t get a ton of traction, so I emailed another 100. I also sent reminder emails a week later.

This seemed to do the trick: I ended up with 28 completed surveys.

Data analysis and standardizing categories

After skimming the results to see how they compared with my initial thoughts on categories, I dug into the categories users created. To simplify my analysis, I set about standardizing categories. In the category tab, I looked through the list for categories with the same or similar names. If they were similar, I opened them up and checked that they held similar resources. When I saw at least 3 similar resources, I would standardize the categories. A nice way to ease into this was by standardizing the ‘low hanging fruit.’ I standardized categories like ‘classroom management’ and ‘curriculum’ because the content almost all matched up.

As I was going through this first pass I realized I had a ton of unnamed categories. Less than ideal. I’d allowed people to leave categories unnamed because I didn’t want anyone to feel stuck and abandon the survey. But next time I won’t let people leave categories unnamed because it makes for messy data analysis: I’m not even sure sorting through the unnamed cateogories was worth it!

After the easier (more obvious) stuff, I dug into the harder to standardize categories. The word ‘collaboration’ came up a lot, but there wasn’t a ton of consistency in the cards within the categories. Here’s where the similarity matrix came in handy. I’d look at the cards under collaboration, find them in the the similarity matrix and see what they most often matched with. I just kept going back and forth between the two tabs to try to understand what people meant when they grouped things and how often certain cards grouped together.

As I did this with ‘collaboration’ I realized that it fit much better under a bigger category of School Culture. What’s great about this is now I potentially have a category AND one of its subcategories. I kept going in this fashion until I had almost all the categories standardized. Samantha said she is a purist and likes to get everything standardized, so I did my very best to do that.

There were a few challenges to standardizing everything. There were often outliers or categories that just didn’t make a ton of sense to me. When things were one-off and not named, I added them to an outliers category. I looked at the outliers and asked myself if they had any commonalities, if there was a labeling problem, or if they just didn’t fit with our content strategy.

Sometimes you could look at a grouping and see the logic behind what the person was thinking, but it just didn’t standardize with others. This just shows you how different users think differently about our content, and gives us alternate ways to categorize in the future. There were some things I was able to move out of outliers after finding patterns, and some just hung out there.

Once everything was standardized (or shoved into outliers…) I moved to keynote, I pulled out all my standardized categories and did my own bit of card sorting until I came up with 4 larger categories, each with some subcategories. Yes, our new taxonomy will need to be tested, and the outliers dealt with. But for now, I’m feeling pretty satisfied with the result.

Three useful reflections

Every study I run, no matter how confident I am in my approach, leaves me new insights into how to improve on the next study, or what I need to continually remind myself.  I also always come away knowing something I didn’t about my intended audience.

Here’s the three I’m taking away from this card sort.

Good setup is incredibly important for easier data analysis

Heading into this card sort, I took with me what I’d learn from the previous one: make sure the cards are simple to understand so that participants don’t get overwhelmed, and general enough so that no card is impossible to categorize with others. And after this card sort, I have a few new things to add to take away: that ‘leadership trap’ I mentioned was real, even when I tried changing words around. And allowing people to leave categories unnamed created a whole lot more work for me standardizing categories.

I’m grateful that I can take these tips into my next study (which I’m sure will teach me even more!).

Be careful of your own personal bias

I will admit it. There were times when I looked at the data and thought ‘What?! These people are wrong’, or ‘They just don’t understand the exercise.’ I probably shouldn’t admit this publicly when writing about user research! But once I got over these moments (and put myself right), I was able to see nuances behind how people categorized the content, and therefore how they think about these resources and their jobs as administrators.

School principals and vice principals juggle A LOT

They do everything from supporting teachers to overseeing playground safety, from managing parents to juggling budgets. For the resources to be useful, the taxonomy we design has to be simple enough for people to find what they need, but complex enough to handle all the different facets of a school administrator’s job.

So, what next?

While I’m pretty happy with the results of this open sort, I’m not 100% ready to put this on our site tomorrow. To start with, I think it would also be great to get on the phone with a few users to do either a heuristic type test — ‘How does this look to you? Are we missing anything?” — as well as some task-based tests with a prototype. And I plan on running a tree test to establish how easy it is for our intended audience to navigate the taxonomy while completing common tasks.

I may even write these studies up as well.

All fun things to come!