All grown up: what first-click testing can teach us about an enterprise

Adventure must start with running away from home.

— William Bolitho

Large enterprises are wonderfully complex creatures.

Enterprises change organically over time. They get together; they break up—sometimes amicably, but not always; they make new friends; they spawn smaller companies; they rebrand themselves.

And sometimes an enterprise will evolve to the point that it needs to pack up and move out of home.

Time Warner Cable (TWC) is one such example: after years of growth and diversification, the company had reached the point where management agreed it would be better off separated from its parent, Time Warner Inc. On March 12th, 2009, that separation became official.

Almost six years on, TWC is now the second-largest cable provider in the US, ranking at #134 on the 2014 Fortune 500 list (not far behind Time Warner Inc at #102).

By the numbers, both organisations are performing admirably. And they both clearly have good taste, as indicated by their mutual love of Game of Thrones (just sayin’). However, I was curious to see how the user experience was faring since the separation, and decided to test both homepages using Chalkmark.

Setting up the studies

Chalkmark is a first-click remote testing tool. It reveals the first impressions that your users have of designs and screenshots. You can use this information to determine whether your users are heading down the right path when they first arrive at your site.

I’m a fan because the tool fits quite nicely with the ‘test early, test often’ approach that I like to follow. Testing early and often allows you to gain quick insights from your users before you invest time and money in design changes.

For my research I decided to run two separate Chalkmark studies—one for each website. For consistency, both tests consisted of seven tasks and post-survey questions that were unique to each site.

Post survey questionnaires

Chalkmark, just like its siblings OptimalSort and Treejack, includes the handy option of asking participants pre- and post-survey questionnaires. The questions you ask can be either quantitative or qualitative in nature, and are an opportunity to gather some bonus insights from your participants.

For TWC, participants were asked four questions post-survey:

• Do you have a Time Warner Cable subscription? (Multiple choice, single answer radio select)
• If you don’t have a Time Warner Cable subscription, why not? (Multiple line text answer)
• If you do have a Time Warner Cable subscription, what do you think they could do better? (Multiple line text answer)
• Which of the services offered by Time Warner Cable are you aware of? (Multiple choice, multiple answer checkbox select)

For Time Warner Inc, I asked participants just the one question after the survey:

• If Time Warner Inc was a person, what would they be like?

I’d never asked this question before—I was curious to see how participants would feel about the organisation and what they would say when asked to apply human-like qualities to it.

Launching and sourcing participants

Like the rest of the Optimal Workshop suite of tools, building and launching a Chalkmark study is quick and easy. Chalkmark also allows you to set closing rules, and for both tests I set a participant limit of 30. This means that the test will close automatically after 30 completed responses have been received. However, anyone who clicked on the link for the test before it closed will still be allowed to complete their test, even if the limit has been reached. Think of this as a rule for ensuring you have a minimum number of participants, rather than a hard limit.

As soon as I launched both studies, I handed the participant recruitment responsibilities over to the Optimal Workshop team. I could have recruited participants myself by tweeting or emailing the activity link out to my network, but I wanted to be sure my specific screening criteria would be met. I used the same criteria for both studies—I requested that participants:

• currently reside in the US,
• are an equal mix of males and females, and
• come from a diverse mix of backgrounds, occupations and ages.

If you’re interested, you can read more about recruiting participants in this article.

So what happened? Let’s look at the results.

TWC Task Results

Overall the TWC homepage tested quite well.

50 participants completed the TWC study. Of those 50 people, 37 people completed all seven tasks, while 13 people abandoned it.

The first thing that really jumped out at me when I looked at the TWC website was the number of possible ways to access information about the services available. I knew where I would start, but I was very curious to see how others would approach this. So I presented participants with the following task:

You don’t currently have a cable TV subscription, but you’re thinking about getting one and you’re weighing up your options. Where would you go to find out what Time Warner Cable is offering?

This task tested very well, with only three participants starting off on the wrong click. The 34 participants who did take the right path approached the task in a number of ways:

• The majority of clicks went to Plans & Packages on the global navigation bar (46%)
• In second place was View All Plans & Packages on a (non-moving version of) the carousel (20%).

The remaining clicks were quite scattered (although none of them were incorrect!) across Watch TV, Discover TWC, Shop Offers, TWC ID, Enter ZIP and Search. I found it very interesting to see how participants assessed their options when shopping around for Cable TV—some headed straight for the pricing options, while others would first check whether the service was even available in their area.

Where would you go?

Most of the participants in this study had no trouble identifying where they would go to pay their monthly bill.

• 66% of participants spotted the handy Pay Bill link at the top of the page, while 17% headed for My Account
• Most participants in this study were easily able to find their way to technical support when presented with the scenario of a pesky modem that wouldn’t play ball: 77% selected Support on the global navigation bar.
• When asked where they would go to ask a question about their service, 59% indicated they would also head for that same Support link, while 24% clicked Email at the top of the page.

The Support section is where a lot of the self-help materials are housed and these results indicate that the TWC homepage is doing well to guide users into self-service behaviours. Whether they’re able to find what they’re looking for is a whole other story but supporting your users to troubleshoot basic issues on their own is a really good thing. Not only can it help to ease pressure on your client contact centres, but it can often result in a better user experience if users are able to solve their issue quickly and get back to doing what really matters.

When presented with the scenario of moving house and needing to find information on what to do about an existing cable service, a total of 66% of participants were able to do this successfully:

• 25% clicked on My Account
• 33% went to Support, and
• 8% clicked on TWC Central at the very top of the page.

I also asked participants where they would go to locate a TV Guide.

• The majority of them were able to find it, with 64% selecting ‘Watch TV’.
• Of the remaining participants, the results were a bit scattered across the page with clicks recorded on: the Time Warner Cable logo, About Us, Discover TWC, Plans & Packages on both the global navigation bar and the stationary carousel, and Shop Offers. In addition to this, one participant clicked on Search and another headed for the Support It was clear that some participants were confused, but I wouldn’t call this one a showstopper.

Lastly for TWC, I was curious to see if participants would be able to locate one of the other services offered. As the name suggests, TWC is a cable TV provider, but they also provide internet, phone and home security products and services. I asked participants:

Where would you go to learn more about Time Warner Cable’s home security services?

The results were interesting. With only 34% of participants starting on the right track, the results on this one were mixed and clearly laced with confusion. Of the remaining participants:

• 9% would search for the information
• 9% selected My Account
• 11% selected TWC Central
• 6% (2 participants) selected Your Home, which is actually a link to the homepage!

Using words like home on a website can certainly help people find their way back to the homepage. However, in this case the phrase Your Home links to subject matter related to the bricks-and-mortar variety of home, and is positioned right next to the logo. While the numbers may not seem significant, given that only two people clicked it, the risk for potential confusion is clearly high and, I believe, warrants further investigation.

Overall, the homepage of TWC tested quite well. I think this is a really positive discovery—it’s just as important to know what’s working for you, so you can focus your resourcing efforts on what isn’t quite there yet!

Time Warner Inc Task Results

40 people participated in the Time Warner Inc homepage study. Of those 32 people completed all seven tasks and 8 abandoned it. This study surfaced some potential opportunities for improvement. While the abandonment rate was lower than that observed in the TWC study, there were some tasks that resulted in participants clicking randomly on the page—possibly because they got stuck. Let’s dive in.

The Time Warner Inc homepage at the time of this study was made up of a collection of large images that look like clickable tiles—they’re not. Each tile (let’s call them that, to make things easy) has a text label that serves as the link, and you actually have to click on the words to visit the link destination.

I was curious whether this distinction would be clear to participants, and for 54% of them it was. While that may sound like a good result, it’s also true that 31% of participants clicked on an image that would take them absolutely nowhere. Imagine how much time could be saved for users if the entire image was coded as a link? Food for thought.

I also asked participants where they would go to learn more about Time Warner as an organisation.

• When asked what the company does, 51% selected Company on the navigation at the top of the page.
• 77% of participants were able to work out how to apply for a job with Time Warner Inc from the first click.
• 57% were able to locate a press release.
• Most participants (73%) in this study also had no trouble locating information on the performance of Time Warner Inc stocks; just over half (52%) selected Investor Relations at the top of the page.

The Time Warner Inc homepage has content filtering options (in light grey text) that allow the user to decide between seeing brands or categories. However, I found these filtering options a little confusing.

• The filter is set to Brands by default.
• To filter the content by a specific brand (e.g. HBO), you need to click on a small circle beside the name of the brand, not the HBO text label next to it.
• To switch to a categories view, you select Categories from a dropdown menu, and repeat the click-on-the-small-circle interaction for each category.

Predictably, the participants in this study also found this interaction to be counter-intuitive, and it clearly has room for improvement.

Participants were also asked:

You’d like to find out more about the TV shows that you can watch on HBO. Where would you go to do this?

• 52% of participants clicked on the text label HBO (which, as mentioned earlier, is not actually a link).
• Just one participant clicked on the small-circle-filter-thing.
• One participant clicked on the dropdown beside the label Brands (potentially expecting to see a list of brands from which to choose).

A close-up snippet showing the selections (participants were shown the same homepage image for all tasks)

Besides the obvious colour contrast issue of light-grey text on a white background, these results demonstrate that the filter tool just doesn’t make sense to users. A quick-and-easy fix could be to make the text labels clickable. This would make the design more forgiving, and less frustrating for users.

The last question I asked participants in the Time Warner Inc study was where they would expect to find an article describing a “behind the scenes” look at an episode of Game of Thrones. Only one participant found the article!

That awesome content happens to live on the Time Warner Inc blog. However, 35% of participants in this study did click on the filter label HBO, which we know isn’t even a link—although it’s nice to see that more than a third of participants clearly associate the award-winning TV series with HBO.

Post-survey questionnaire results

TWC results

The post survey questionnaire for TWC revealed that 86.5% of the 37 participants (32 participants) did not have a TWC subscription. When asked why they didn’t, the responses received were very interesting:

• 34% of the 32 said that the service is not offered in their area
• another 34% were with another provider
• 22% didn’t want or need the service, and
• the remaining 10% couldn’t afford it.

I completely understand that not all services are available in all geographical locations, but I think the 34% who are with other providers and the 22% who stated that they weren’t interested are worth a closer look.

Here’s a sample of some of the comments:

• Because time warner is not available in Anchorage, Alaska.
• We do not have cable at all. We rarely watch TV.
• I have a different cable company that I am reasonably satisfied with.
• I am a professional dancer and can’t afford one.

The 3 participants who do have a TWC subscription were asked to share what they thought could be improved; their responses were all different. The suggestions were:

• Make it cheaper
• Make their internet faster
• Stream online

Questions like this are a great forum for people to share their ideas, and you never know what they’ll come up with!

The fourth and final question asked of participants in the TWC study was designed to gauge their level of awareness around TWC’s service offering. The answers were quite positive:

• 4% of participants were aware that TWC was a TV provider,
• 6% were aware of the internet services on offer,
• Only 27% of participants were aware of the home security services, and
• 8% selected that they were not aware of any of the services listed.

From this date it is clear that there are opportunities for TWC to boost awareness with their consumers about these services.

Time Warner Inc

Participants of the Time Warner Inc study were asked to describe what sort of person Time Warner Inc would be. The responses received were a real mixed bag!

The question was listed as optional and it was wonderful to see that all 32 participants shared their thoughts. Their responses ranged from “quite positive” all the way to the other end of the scale of “not even remotely positive”. About half of the responses sat somewhere in the middle.

Here are a few quotes taken directly from the responses:

The positive

• A kindly old grandfather.
• Funny, long lasting and rich plus well rounded in a lot of different areas.
• They would be a very reliable yet innovative person.

The not-so-positive

• They would be greedy, incompetent. They would believe they were one of the best, but really be one of the worst people.
• A horrible person

The neutral

• They would be pretty chill, but also cost a lot of money to take out to dinner
• The kid that always watched all the movies
• They would have a lot of personalities because they have so many different programs and channels.

You’ll remember that I was curious to see how people would respond when asked to apply human qualities to Time Warner Cable. The quality of these responses show that this is a really interesting technique for gauging how your users feel about the organisation overall.

Home is where the heart is

Both studies produced interesting (and mostly positive) results. The mix of qualitative and quantitative data paint a picture of what these two organisations are doing well in terms of the experience they offer their users.

And while this research also uncovered a few areas needing some improvement, it’s important to remember that no design is truly ever finished or complete—there are always elements of a website that can be improved. TWC has not only moved out of home, but has clearly reached that level of functional independence where it’s no longer coming around once a week to use the washing machine or raid the fridge.

I thoroughly enjoyed performing this research, and would love to hear your thoughts in the comments below.