Teacher taxonomy: What tree testing told us about our website

At Bloomboard, we’ve been working on a project to improve our online resources for the K-12 teachers and administators who use our platform. We already have in place a comprehensive ‘Educator Resources’ site aimed at teachers, and we’re creating a similar site for administrators (the principals and managers who run the schools).

I recently ran two remote studies for this project, and I’ve written both up as short research stories (for your interest and joy!).

• Part 1 describes the tree test I ran to assess the findability of content on the existing ‘Educator Resources’ site.
• Part 2 describes the open card sort I ran to generate ideas for the structure and content of the new ‘Administrator Resources’ site.

What follows is Part 1 on tree testing.

Why we ran a tree test

We wanted to test the findability of our ‘Educator Resources’ content based on common user tasks. We knew anecdotally that teachers visited the site and found it useful, but we wanted some quantititave data to answer two main questions:

1. Do people understand the difference between the top-level labels, ‘Skills I’m using’ & ‘Skills I’m teaching’?
2. What categories or sub-categories do we need to tweak or change to make the content clearer?

Creating our tree from the site map

The site currently contains two different categories of content:

• Skills I’m using: Teachers can use this content to inform their own professional development, and to improve their teaching practice.
• Skills I’m teaching: Teachers can use this content as teaching material in the classroom to improve student learning.

As I was setting up the tree, I questioned whether I should include all the subcategories, because I worried that 60 subcategories might overwhelm our testers (I’ve since learned that this is quite a small tree!). However, my mentor Samantha Bailey (yep, we have the same last name) encouraged me to go for it.

In retrospect, I’m happy I did because I discovered that the subcategories caused more issues for folks than the bigger categories.

I uploaded the site map from spreadsheet, so the top three level of headings looked like this:

Writing tasks and selecting correct paths

I wanted to base the tasks on authentic scenarios that might trigger someone to look for a resource. I drew as much I could on my past teaching experience and the countless interviews with people I’ve started with ‘So, is there anything in particular you are working on right now?’

Instead of writing literal tasks such as ‘You are looking for a resource to help you with X’ (which felt like giving away the answer), I wrote tasks that outlined classroom scenarios: ‘You are hoping to spur a lively, scholarly conversation about the article your class just read, but want to make sure you give the students the tools to do so. What do you look for?’

Getting the wording right for the tasks took time. I was soliciting all types of teachers from all grades, subjects and skill levels, so the tasks had to be common enough to work for all of them, but specific enough for there to be a ‘clear’ path. I knew this was particularly important because in previous task-based tests, I’d run up against challenges such as when a kindergarten teacher wouldn’t do a lab in her class or when a veteran teacher would never need resources on classroom management. Often when a teacher thinks a task doesn’t apply to them, they lose interest in the rest of the study, and I definitely did not want that to happen!

So that I could gather data on the whole tree, I went through each of the 12 categories under ‘Skills I’m teaching’ and ‘Skills I’m using’, and created a task for one subcategory within each category. Doing this gave me an insight that reflected one of the study findings: many categories and subcategories were similar enough to make selecting the ‘correct’ answer quite challenging — for me! This of course suggested that our audience might struggle with the same problem.

Recruiting participants

I knew I wanted at least 40 people to complete the study, and I wanted them all to be in our intended demographic. The holidays are a crazy time for everyone, and especially educators, so I invited a total of 250 people (!) to complete the study. I also sent a reminder a week after the original invitation. This seemed to do the trick as I got exactly 41 respondents — merry xmas to me!

I also asked pre-survey questions to ensure we gathered data from the right people and to enable us to filter the results based on the answers:

It was great to see that 95.1% of our respondents were teachers, and that they taught at range of levels. And even more exciting was seeing the distribution of teaching experience: having 11 participants in the categories ‘This is my first year!’, ‘2–4 years’, and ’11+ years’ respectively gave me the option of comparing their responses directly (a project for another time, perhaps).

A sneak peak into the two key insights from our results

Once the results came back, I kicked off my analysis by looking for patterns on the task results and pietrees. On first glance, I was worried: ‘Why is their so much red? Is our taxonomy really that bad?!’ But then Samantha helped me dig a little deeper, and I started to see that it actually wasn’t all bad. And what we did discover gave us two great starting points for improving the structure and labelling of the ‘Educator Resources’ site.

Overlapping content and ambiguous labels confused our participants

The two most obvious (and useful) problems we uncovered in our taxonomy were overlapping content (content that could sit in both the ‘Skills I’m using’ and the ‘Skills I’m teaching’ categories) and ambiguous labels for content (labels that misled people into thinking they’d found a correct answer when they hadn’t).

What is the real difference between ‘Feedback from assessments’ & ‘Data analyzed…now what?’. Or between ‘Leveraging the community for your students’ & ‘Engaging the surrounding community’? At one point in the past, when I was knee deep in creating this taxonomy, I could have told you. But right now, I’m not sure I can. And the tree tests have proved that this ambuguity is tripping up our users.

For example, one task said: ‘You want to have career professionals come speak with your students about their job and what it took to get there. You just aren’t sure where to start to find a variation of professionals in your area.’

The results of this task were rather enlightening.

These results show that both the main categories and the content labels caused confusion. Only one person went straight to the correct answer, and one other found the correct answer, but only after backtracking.

The pietree illustrates this further. The centre of the pietree (the homepage) shows that participants went the wrong way from this point 62% of the time.  And of the 21 people who selected ‘Engaging with the surrounding community’ as the correct answer, 100% did so without backtracking (which suggests a level of certainty that they’d found the right answer):

In another task, we discovered something similar. We said: ‘You’ve noticed your students often forget pieces of longer assignments or turn them in late. How can you find resources to help you help your students?’ The pietree illustrates the confusion instantly. The large green line shows the correct path, but we can see that participants went everywhere!

The content labelled ‘Project based learning’ lives both in teacher skills and student skills. Sure, maybe as a teacher I might come at the problem from ‘What can I do to make sure a project runs smoothly in my class’ vs ‘What tricks exists to help my students successfully complete projects?’ But these really are so similar that it’s making it hard for our users (and probably our taggers) to figure out where resources live.

Reflections and next steps

Despite having taught for 3 years, I am now almost 3 years away from the classroom. Doing exercises like this really helps me remember that only by truly understanding the teacher mindset can we categorize in a way that makes sense for our users.

I’m going to share the results of this survey with our Content Curation Manager (another former teacher) to ensure this gels with how he has been tagging resources. He’s mentioned to me in the past that he draws on his past experience, but wants to keep fresh on how teachers are thinking about finding resources for themselves.

One of the challenges for us in creating an intuitive taxonomy is the in-depth nature of the content. We have to read the resource and understand what it’s about, as well as understand what might prompt a teacher to look for it. On top of this, we need to keep a close eye on education industry jargon, and use language to describe things so that our entire audience can understand.

Thanks to this tree test, I think we have a better sense both of how teachers think about looking for things and well as how we can make the taxonomy more accessible to them.