Form review: IRS US Individual Income Tax Return 2016 Form

16 min read Optimal Workshop

Paying tax is one of those necessary experiences that isn’t always positive. Depending on where you live, it could be a single click of a button once a year. Tax time might be on a day of your choosing or it might be something you do on your phone. It can also involve filling out a lengthy form and preparing multiple attachments. For this form review, we’re going to take a look at a first-click study I conducted on an example of the latter. I decided to put the IRS’ US Individual Income Tax Return 2016 form to the test using Chalkmark. This form is completed by millions of people each each year needing to submit a US tax return – in 2016 there were more than 150 million of them processed by the IRS. Chalkmark is Optimal Workshop’s first-click testing tool that measures where that all important first click lands in response to a scenario based task for a website. It also has several other uses and testing a form is just one of them! Let’s take a look at what happened when the IRS met Chalkmark!

Setting up the study in Chalkmark

Form: US Individual Income Tax Return 2016 form

Research scope: Just the above named two page form

Not in scope: Separate instructions, attachments, schedules, the IRS website

Tool: Chalkmark

Just like OptimalSort and Treejack, Chalkmark has the same helpful interface that guides you through the creation process step by step.

Chalkmark works by presenting participants with a flat image and then asking them to respond to a task simply by clicking on that image. The image might be a screenshot, a sketch, a wireframe or something else entirely and the task might relate to a website, an app, a photo or you guessed it, a form. It’s quite a versatile tool!

In this study, the IRS’ US Individual Income Tax Return 2016 form is two A4 pages long and has many different sections. For my screenshot images and tasks, I approached it by the individual sections or by chunking a few together. There’s quite a bit of text on the form which meant I couldn’t just zoom out and screenshot the entire form for each task. That wouldn’t be fair to my participants and large amounts of super small text would have impacted upon the results, so I took screenshots that included multiple consecutive sections without affecting readability and to avoid leading my participants to the correct section.

Chalkmark comes with some really well-designed default messaging and instructions, but for this study, I decided to change them. I just needed to tailor the message to suit the participant audience and reflect the fact that I was researching a form and not a website. When doing research on government experiences, tools and products, it’s important to recognize that participants will sometimes feel a little nervous and perhaps a little distrustful. They often need some extra reassurance that they’re not going to get in trouble because of something they said or did and that their responses are indeed completely anonymous. For this study, I set the participant identifier to anonymous (even from me!) and made the below tweaks to the welcome message to show this.

The customized welcome message in Chalkmark — The changes I made to Chalkmark’s default messaging to suit my audience

Choosing the tasks

When I reviewed this IRS form to extrapolate my task questions, I stumbled a bit. While I worked as a designer for a tax department in the first few years of my career, I’m not American and I’ve never seen this form before — let alone completed it! It was important for me to take some time out to understand it and go through it step by step and see what tripped my fresh eyes up. Ultimately, it would have been even better if I’d had access to some representative users to understand their thoughts, so I could base my tasks on those conversations.

My key takeaways from this high level analysis were:

This form requires a lot of attachments and additional forms
It requires many different types of information to be shared but in different places, e.g., the Social Security Number (SSN) is recorded in a different place to other personal details
There are multiple references to ‘separate instructions’ and ‘see instructions’
This form is completed manually and requires numerous calculations along the way
This form contains a large amount of technical terms that may or may not make sense to users

With this mind, I developed 6 task based questions to help me understand the usability of these takeaways. The tasks for this study were:

You and your wife are submitting separate tax returns this year. Where would you enter her Social Security Number?
You have several additional forms that you need to attach to this form. Where would you place the staple or paper clip?
You moved house this year to start a new job in another state. Where would enter your moving costs on this form?
Where would you enter the total amount of all your taxes for this year?
Where would you enter the amount that the IRS needs to pay you?
You published a novel this year and you’ve made some extra money from it. Where would you include this money on the form?

With Chalkmark, you can include up to 10 task based questions but I chose to include 6 tasks in this study because I wanted to test some of those tax technical terms I uncovered via the post-study questionnaire. With the Optimal Workshop suite of tools, you can add post-task questions (questions that participants will answer after completing a certain task), as well as post-study questions (questions that participants will answer after completing the entire study).

I decided to use a combination of both and set one question to appear after Task 3 and a further 5 questions at the end of the study. I chose to ask my participants‘ This task requires you to attach a form. Where would expect to find that form?’ directly after Task 3 because I wanted them to respond in the context of that task (recording moving expenses). If I’d asked them at the end of the study, they may not have understood what I was talking about! This is one of my favorite things about Chalkmark and I find it really useful for gaining more insights while my participants are still in that specific task headspace.

The 5 post-study questions that appeared in this Chalkmark study were:

What does the term ‘Head of household’ mean to you?
What does the term ‘Third Party Designee’ mean to you?
What does the term ‘dual-status alien’ mean to you?
If you were having trouble filling out this form, where would you go to get help?
Did you prepare your own tax return in 2016?

The first three are tax technical terms that may trip users up and I was curious to hear their thoughts, while the fourth one is designed to help me understand where they would go if they got stuck. Finally, I included the last question to gain an understanding of how many of my participants had actually experienced this form themselves when completing their 2016 tax return. None of these questions were compulsory allowing participants to decide what they were comfortable sharing.

Building the study

Once I had my screenshots, new messaging, tasks and post-study questions, there were a few more jobs to complete before launch.

After pulling it all together in Chalkmark, I needed to set the correct areas on my screenshots. Setting correct areas helps provide a framework for the results — it tells the tool where the right answers are so that it can provide you with hard data on how many participants got it right and how many did not.

The best part about this functionality? You can set or change the correct areas anytime you like! You don’t have to do it prior to launch —you might do it after the study has been completed and is no longer accepting participants! This is particularly handy for when you’re in a rush to get this thing out or when you’re waiting to hear from someone else to tell you where the clicks should fall to be correct. You can do your research while you wait and fill it in later! It’s also great for when you make a mistake or forget about a secondary location that is also correct.

Participant recruitment

For this study, I used the Optimal Workshop recruitment service which can be accessed via the Recruitment tab in Chalkmark. The recruitment brief for this study was 50 participants aged 18 -70 with an equal mix of genders (Male/Female/Other) all residing in the United States.

Results

This Chalkmark study had 57 participants in total and of that number, 50 (87%) completed the activity and the remaining 7 abandoned it. This means that 7 of my participants closed out of the study before they reached the end. It might be that they opened the study and then got distracted or needed to go do something else. A 13% abandonment rate isn’t something to worry about. It’s quite normal to see a handful of abandonments per study and I do have 50 completed responses making for a robust dataset. I’d be concerned if the abandonment rate was higher than 35% as this might indicate ambiguity in the task questions or the study might be taking too much of people’s valuable time. Write your tasks in clear, plain language and include no more than 10 of them per study.

Task results

When analyzing the results from a Chalkmark study, the first thing I do is head to the ‘Task Results’ tab in the results section of my study. This tab gives me high level at a glance insights into how my study performed. For the IRS form, we have a mixed bag. Tasks 1-3 tested somewhat well, while the results from Tasks 4-6 indicate there is some room for improvement. Let’s take a look at one from each camp in more detail: Task 2 and Task 5.

Task 1 results analysis

I included Task 1 (You and your wife are submitting separate tax returns this year. Where would you enter her Social Security Number?) because it was one of the first things I noticed about this form. It asks the user to indicate their filing status in the second section of the form and if they’re married, to enter their spouse’s Social Security Number (SSN) ‘above’. I personally found this really confusing and, no joke, it took me 10 MINUTES to find it! I wondered if maybe it was just me and my fresh first time user eyes which is how it ended up in this study.

Based on the results of this study, it wasn’t just me — 36% of my participants missed the mark too! While it’s great that 64% of participants were correct, it’s a government form and the 36% that didn’t get it could find themselves in hot water if they made a mistake! When it comes to government tools and products, usability needs to be held to a high standard. People don’t have much of a choice — they need to pay tax and we should be seeing a lot more green on the Task pie charts in this study.

As for where they went instead, the Heatmap for Task 1 shows us that 22% of participants would have entered it in the field that asks for their Spouse’s name. An easy mistake to make given the wording of the instructions at that field. Chalkmark’s heatmaps are a really cool visualization tool that show you where the clicks landed and where the clusters were. Based on this one, we can also see that 4% of participants would have entered their spouse’s SSN in the space where they needed to provide their own. Those random clicks that we can see scattered around may indicate that participants weren’t sure where to go.

Task 5 results analysis

Correctly entering the amount that the IRS owes you as a tax refund is pretty important. In this Chalkmark study, 58% of participants failed to locate the correct field, 1 person (2%) skipped the task while just 40% were successful. The average time taken to complete this task was more than 20 seconds (24 seconds) which is quite a long time in terms of task completion. When testing a website, most task times that I’ve seen tend to fall between 5 and 15 seconds and generally speaking anything over 20 seconds is a red flag. Keeping in mind that I was testing a complex government form, this is probably OK because people will often take more time to read over everything and double-check their answer before submitting!

The most interesting thing about this task is when I dig into the Heatmap, everything appears fine (see below).

Task 5 Heatmap in Chalkmark — Task 5 Heatmap in Chalkmar

Underneath that cluster that has 71% of the clicks in this task lies the correct field — the one marked ‘76a’. The problem is, there’s actually 3 narrow fields under that cluster (74, 75 and 76a) and from here it is hard to tell exactly where those clicks fell. Chalkmark records every click that was made on the images during the study and it shows them to you via a handy drop down option just above the heatmap visualization (see below).

Heatmap visualization options in Chalkmark

If we click on that and choose ‘Selection’, we can see what happened (see below).

Selection map visualization in Chalkmark

If I zoom in nice and close like I have below, I can see that more than 10 people would expect to enter the amount owed to them in the field marked ‘75’ which is actually used to record the result of a calculation you use to determine how much you overpaid in tax. The amount owed and the amount overpaid, aren’t necessarily going to be the same number — it wasn’t a trick question I can assure you! I wanted to see how the instructions for those two fields would test and see if those instructions might trip people up. Furthermore, looking back at the screenshot above, there are a lot of scattered clicks for this task that have landed squarely in other fields — more than one in some cases! This suggests that these were indeed legitimate attempts to complete the task, rather than random clicking, and overall people seemed lost.

The same Selection map visualization in Chalkmark as above but zoomed in

Other findings from this Chalkmark study

62% of participants in this study failed to correctly locate the field in which they would enter the total amount of all their taxes for the year. Of the total number, 34% of participants would enter it in the field marked ‘74’ which is where total payments should be recorded
58% of participants were unable to determine where they would record income earned through the sale of a book they had written during the year.
10% of participants in this study did not notice the instructions around where not to attach staples
82% of participants were able to correctly identify where they would enter moving costs for the year on this form

Post-study questions

For this Chalkmark study, we had 5 questions at the very of the study and 1 question that appeared immediately after Task 3.

Remember, Task 3 required participants to locate where they would enter moving costs. When I was building the study, I noticed that the instructions for this particular field referenced a form that needed to be attached, so I thought it would be interesting to immediately follow that with the question of: ‘This task requires you to attach a form. Where would expect to find that form?’. If I look at the Task Results for Task 3, I can see a link to participant responses in the bottom left-hand corner (see below).

Access the task questions via that link highlighted in yellow in the bottom left-hand corner

This question received 49 responses — that is excellent given that answering it was non-compulsory! Furthermore,30 of those responses indicated that they would expect to find the form on the IRS website, which is of course correct!

Moving on to the results of the 5 post-study questions that appeared at the very end of this Chalkmark study, I received some interesting responses. The first three were designed to see how well participants understood some of the tax technical terms used on the form.

First up was ‘What does the term ‘Head of household’ mean to you?’ and I’m dismayed to say that quite a few of the responses looked like this one: “the man of the home”. Amongst the rampant #Everdaysexism I observed, I found that only 1 of the 50 responses to this question was correct and a further 2 participants came close but were missing key information. Everyone else was way off base on this one; that’s 47 people who didn’t understand a term used on an essential government form! The legal definition of ‘Head of household’ for taxation purposes in the US isn’t as straightforward as it appears and it requires taxpayers to satisfy multiple criterion and there are also a number of special rules that apply. It is not as simple as men who make money and pay bills, and people who are thinking that while completing this form could lead to errors.

Next, I asked my participants ‘What does the term ‘Third Party Designee’ mean to you?’. Out of the 50 responses received, only 12 people were correct. A Third Party Designee is someone you authorize to discuss the content of this form with the IRS on your behalf and the form field is labeled as “Do you want to allow another person to discuss this return with the IRS (see instructions)?”. It’s about giving another person permission to discuss your private tax information with the IRS. In this study 24 participants had a red hot go at deciphering the meaning of the term but were incorrect. Their responses listed a variety of possibilities including: tax preparers, businesses, people that didn’t live with them and disturbingly one participant felt it was ‘The person who comes second for priorities over the house.’ The remaining 14 indicated that they were unsure. Participant 37 summed it up nicely when they said “I don’t know, I would have to look this up.”

The third question I asked was ‘What does the term ‘dual-status alien’ mean to you?’. A dual-status alien is someone who has been a non-resident alien and a resident alien for tax purposes in the same tax year. It usually refers to someone who has recently moved to the US or away from it after living there as a resident alien. It’s just a tax thing and it has nothing to do with citizenship status. In this study only 2 of the 49 responses received to this question were correct. Of the remaining 47, a total 37 participants commented that it was related to citizenship and there were several references to the legalities of that citizenship. The form itself doesn’t reference any further guidance or instruction on how to determine if someone is a dual-status alien. It seems to assume that if you are one, you know, but that doesn’t help people who don’t know.

If they were having trouble filling out this form, 29 of the 50 participants in this study would head straight to the IRS website. Other responses included ‘call my parents’, ‘Google’ and ‘Ask a professional’.

And lastly, I asked my participants if they prepared their own tax return in the year that this form came from. A total 48% of participants in this study indicated that they had. It is possible that they didn’t all fill out this form as there are many different ways to file a US tax return and in hindsight I probably should have asked if they used this particular form (lesson learned). Given the results from this study – especially the field related issues and the level of understanding of the terms used — I do find it concerning that almost half of these people are doing it themselves. Based solely on what I’ve seen in this study, I do wonder if they are getting all the support they need and deserve to carry out this necessary irritation of a task.

Ultimately, forms like these shouldn’t exist. There are better ways for people to interact with government but I understand that these things take time. Below are some key findings and recommendations to improve this form in its current state.