Treejack takes a trip with American Airlines

Airline travel can be a truly magical experience. Whether it’s for business or leisure, the time spent above the cottony fluff of the clouds feels like another world away and don’t you just love it how it’s always sunny up there?

On the flip side, airline travel can also be a complete nightmare. Trapped in a cramped metal cylinder with a bunch of strangers for several hours can turn ugly really fast. The fight for cabin baggage space, the projectile vomiting infant (true story!) and don’t even get me started on the passenger with the thimble sized bladder who prefers to be seated by the window. All of these have the makings of a pretty miserable day in the skies.

Whether we have a good experience or not, our one constant companion on that journey is the airline. From pondering the possibilities for our next vacation all the way to hauling our suitcases off the carousel before going home, they are right there with us and they have an enormous amount of influence over that experience. With 57% of all travel reservations made online and all the options that go with that, airlines need to ensure their digital channels are delivering a great user experience. The only way to find out exactly how well they are doing this is to test it and I thought who better to put to the test than the world’s largest airline? Fortune 500 company American Airlines Group became the world’s largest airline in 2013 when it acquired US Airways.

So, what happened when Treejack took a trip with the American Airlines website? It’s time to sit back, ensure your arm rests and tray tables are in an upright and locked position and get ready for takeoff.

Fueling up the aircraft: Setting up the study

Treejack is an online information architecture (IA) validation tool and setting up a study is quick and easy to do. The first step is to build out your IA in a simple spreadsheet and bulk import it across into Treejack to create your tree.

Once I had the American Airlines website IA ready to go, my next job was to create the scenario-based tasks for my participants to complete during the study. The American Airlines IA is quite large and went down to five layers so I decided to use seven tasks to make the overall completion time a bit easier on my participants. There is no limit to how many tasks you can include in a Treejack study, but you have to consider the needs and time constraints of the person completing it. If there are too many tasks, you may find an increase in the abandonment rate for your study. I recommend using no more than 10 tasks for a Treejack study. The American Airlines website uses tabs across the top, quick links in the middle of the page and some really useful content is stored in the footer. With this in mind, the seven tasks were developed to cover as many of the different areas as possible. My third job in the setup process was to set the correct answers to my tasks. This is how Treejack measures success and these answers were based on where the information currently sits on the American Airlines website.

Treejack also provides the option to include pre- and post-study questionnaires. These are like survey questions and are incredibly useful for gathering information you may not already have about your user, or for gaining deeper insights. For this study I decided to include three pre-study questions and one post-study question.

Lastly, I needed to decide how my participants would be identified (I always set this to anonymous) and that jet was ready to taxi down the tarmac.

Boarding the passengers: Participant recruitment

Recruiting participants for a Treejack study is a lot easier than it sounds and can be done in no time at all. All you have to do is create a list of attributes you’d like your participants to have, fill out the form and Optimal Workshop takes care of the rest. For this study I asked for 30 participants, all of whom reside in the US, and it was all done in less than the time it takes to fly from LA to New York.

In flight reading material: Interpreting the results

Making sense of the results from a Treejack study is easy and the tool actually does most of the work for you.

Simply click into the results section of your study as listed in the Treejack dashboard and you will find everything you need organized neatly under a series of tabs (above) under “Analysis”:

• Participants: This is where you will find a detailed list of all participants that started your study. If you set your participant identifier to anonymous, Treejack will assign a number to them, which is useful for tracking individual movements across the entire study. This tab allows you to apply filtering options and displays the following by participant: time taken to complete the study, if they skipped any tasks, how many tasks they completed, how many of those completions were successful and how many of the pre- and post- study questions they answered.
• Questionnaire: If you included pre- and post-study questions in your study, you will find the results from those here.
• Task results: A high level task-by-task statistical breakdown is readily available here to provide you with solid quantitative data that you can present to your clients and stakeholders. This is also where you will find the pietrees. Pietrees are comprehensive pathway maps that not only allow you to see the overall picture at a glance, but are also interactive so you can drill down to a deeper level of granularity.
• First click: This is where you can find out if your participants started their journey on the right foot from the very first click. This functionality works just like Chalkmark but without the visuals.  While this certainly doesn’t replace running a Chalkmark study on your design, it does help complete the picture of your Treejack results.
• Paths: This tab displays the complete details of the pathways followed by each individual participant broken down by task. It shows every single click made in the order it was made in and also displays whether the participant was navigating forwards or backwards through the IA.
• Destinations: A matrix showing where your participants ended up with the IA on the left and the task numbers along the top. Where applicable, each cross-section square shows how many participants ended up at that location for that task and is shaded in either red or green to show whether or not this answer was correct.

Task results:The sunny side of the study

This Treejack study had 34 participants and of that number, 30 completed the activity and four abandoned it.

Overall, the results from this study are a mixed bag of things that are working well and things that could use some improvement. This can easily be seen by looking at the handy “Tasks” bar graph (below) in the overview tab.

Let’s start with what went well in this study!

• 60% of participants in this study were able to locate information regarding airline restrictions on flying while pregnant.

• 77% were able to find information on purchasing a gift card. This was a surprise! This information is located in the footer but clearly its position in the IA is working well for users of the American Airlines website.

Task results: Turbulence ahead

Now let’s have a look at the results that highlighted a few areas for improvement.

• 90% of participants in this study were unable to find information on the process for locating missing or lost baggage. Imagine you’ve just stepped off a long haul flight and after standing at the baggage carousel for more than half an hour, your suitcase is nowhere to be seen and the line for the service desk is huge. What do you do? You whip out your smart phone and see if you can find out for yourself. The American Airlines website has a bag tracker tool that allows passengers to search for their baggage if it has been missing for a certain period of time. This tracker tool, if utilized effectively and positioned better in the IA, has potential to reduce burden on other help channels such as call centres and airport help desk staff. The pietree (below) for this task is a little scattered and the blue sections of the junction pie charts show where participants were navigating back and forth through the structure trying to locate the information.

• When asked where they would go to purchase return airfares, 63% were unable to find the correct location. Many participants were able to find their way to the “Flights” or “Find flights” junctions, but as shown in the pietree below, went the wrong way at the last turn when presented with “Round-Trip” and “One-Way”. Participants were in the right place in the IA but the language used for the labeling appears to have caused some confusion.

• 67% of participants in this study were unable to locate an existing rental car booking for the purposes of updating their details.
• 60% were unable to find information that would help them to upgrade a flight using miles.
• 57% were unable to locate information that would provide guidance on having their boarding pass sent to their phone. American Airlines do have a mobile app for their passengers, which wasn’t part of this study, and that information is available there.

Questionnaire results

The pre- and post-study questionnaires used in this study produced some interesting results. The three pre-study questions were:

• Are you planning to travel in the next 12 months? (Multiple choice, single answer)
• Have you traveled by plane within the last 12 months? (Multiple choice, single answer)
• If you answered yes to the previous question, which airline did you fly with? (Multiple line text answer)

I asked participants the first question to gauge how many were currently considering and planning travel. Questions like these are a good opportunity to gain insights in what your users are thinking about in their lives and potentially help identify future users of a product or service. I felt it was important to distinguish the vacation travelers from the business travelers because a vacation is a choice and the same can’t always be said for business trips! As it turned out, 70% of participants in this study were planning to take a vacation in the next 12 months.

Next I asked about recent airline travel experience. I was curious to see how many of my participants had recently travelled by plane and 63.3% had (19 people).

This question served as a precursor to the next one, where I asked those who said yes to name the airline they traveled with. Of the 19 participants that had flown in the last year, 18 of them responded to question three and the results were:

• United: 3
• American Airlines: 4
• JetBlue: 1
• Delta: 4
• Southwest:3
• Spirit: 1
• Frontier: 1
• Eva Airlines: 1

American Airlines scored an equal tie with Delta and Southwest and United wasn’t too far behind.

Lastly, this Treejack study concluded with one post-study question: What does your dream vacation look like? (Multiple line text answer). I included this because participation in a research study is like any other experience and I wanted to go out on a high note. Questions like these are also useful for providing a glimpse into what your users are interested in and what they look for in their ultimate vacation experience. The responses to this one were all positive and here are some of the highlights:

Participant 3: A week in Japan, visiting both urban and rural areas.

Participant 22: I want to go on a cruise around the world. To stop at almost every continent and get a little piece of each place to see where I’d want to travel next.

Participant 26: I want to go somewhere with a beach. I would stay in a beachfront hotel and spend my days on the beach and in the water. There would be nice restaurants in the hotel and in the area where I could go to enjoy great dinners. At night there would be clubs and other nightlife nearby where I could go to dance, drink, and listen to music. After that I would retire to my hotel room and enjoy a nice night’s sleep before doing it all again the next day. I would invite a few of my friends along and we would have the time of our lives.

Thank you for flying with us today — we have now reached our destination. We hope you had a pleasant journey and hope to see you again soon!