Usability Test Plan: External Link Indicators

Created on 4/23/21

Last updated on 7/31/21 by Sofya Savkina

1. Background

We updated our Link guidance in February 2021 based on research from the USWDS Accelerator project. After updating the Federal Web Council, we received some useful feedback 🔒. We addressed most of this feedback with guidance copy edits, but a few items required further UX research and design exploration. We wanted to dig into how we address the guidance from M-17-06 policy:

 "Agencies must clearly identify external links from their websites. [...] Agencies should choose the best approach to identify external links to users in a way that minimizes the impact on the usability of their websites and digital services."

Our link guidance instructed agencies to omit the share icon (a rectangle enclosing an arrow pointing to the upper-right corner) and use text content and explicit metadata to indicate link destination. This was based on our own experience and on gov.uk research that shows people often don't understand the meaning of the external link icon.

Since M-17-06 policy instructs teams to "clearly identify external links" we wanted to better understand how different approaches to communicating external links affects usability and comprehension. USWDS guidance advises teams to make the link's destination clear in the link text but we wanted to understand which, if any, visual indicator best supplements this approach. We decided to conduct qualitative usability testing to evaluate our options.

2. Goals

During this round of comparison usability testing, the team aimed to understand the impact of the external link indicator on user behavior and experience with reading content on government websites. 

This study helped to measure the effectiveness of each of the two external links' indicator options and learn whether adding the indicator was enough for users to recognize the difference between the internal and external links on the web page. In particular, we wanted to:

• Validate or invalidate hypotheses by testing two variants of external link indicators and comparing their performance.

• Learn if users anticipate that by clicking on the external link, they will be taken out of the current website to another website with different navigation, user interface, and security levels.

• Determine whether  making external links visually distinct from internal links is enough to signal that the user will be taken to another website.

3. Questions

The visual design and guidance for the new external link indicator were based on the hypothesis that the meaning of the existing external link icon is not clear to the users (i.e., it's frequently confused with the "share" icon). An "EXIT" badge indicator might be a clearer alternative. 

With this study, we aimed to validate or invalidate this hypothesis and answer the following research questions: 

• Which identification option is more clear and usable: the existing external link indicator (A), or the newly designed "EXIT" badge (B)? 

• Does the addition of  one of these two external link indicators to text make a clearer distinction from internal links? 

• How do users interact with links?  Do they understand that it's a clickable element? Do they click on the text link or on the icon? Do they review content on the page first before clicking the links, or do they click the links right away?

4. Method: qualitative comparison A/B testing

To determine which visual option is most understood by users as an indication of an external link, we conducted a qualitative comparison usability testing of two variants. Each variant was incorporated into a web page with real content, one variant per prototype. The body copy on each prototyped page was exactly the same---the only difference was the visual treatment of external links.

Variant A featured an existing indicator icon

Variant B featured a newly designed "EXIT" badge indicator

Prototype

We followed the updated USWDS guidance and, using a real web page of the stopbullying.gov website that featured  both internal and external links, we designed two versions of a clickable inVision prototype for this study.

We tested version A of the prototype with one set of users and version B prototype with another set of users, to see which option performs better and is more clear to users. We did not show both versions to the same set of users, so as not to skew testing results.

Session details

• Six remote, 35-60 minute recorded sessions via Zoom, using screen-share functionality: 

• Three sessions for variant A

• Three sessions for variant B

• The participants were given a scenario and asked to review the page mock-up. Then, they completed three tasks outlined in the discussion guide section below. We observed their interactions and questions and then probed with contextual follow-up questions.

5. Participants

The experience with external links on government websites pertains to a wide range of audiences. Since external link indicators will potentially be implemented on numerous websites, we needed to see if our solution is clear to the majority of government services' visitors, across different levels of comfort with technology. The problem we were trying to solve impacts low-tech users the most. If we get it right for the people who feel the problem the most, we'll get it right for the majority. Therefore, we prioritized recruiting people who are: 

• Members of the general public with low levels of technology comfort, such as seniors, folks who don't work on/with computers daily, but use computers for their personal needs.

User recruitment

Due to the absence of a budget for incentives, the team reached out to friends and family networks to recruit participants for this study. This recruitment technique was highly dependent on participants' availability, which impacted the timeline of the study. This risk was communicated to the product owner prior and then  assumed by the team.

Recruitment Communication Protocol

The following TEMPLATE language was used to recruit participants:

1st email: Request to participate 

Subject: Help us design intuitive government websites

Body:  

Hi [Name],

My name is Sofya, I am a user experience researcher at the General Services Administration. Our team is conducting research to learn how people read and navigate the web. 

We are very grateful for your participation in this study. Would it be possible to set up a 30 min video call with you next week to test our prototype and get your feedback? Your responses will be anonymous and you will not be asked to share your personal information.

Just to give you more context, this session will be done via a Zoom call. You will need to join from your computer and be able to share your screen with us, so that we can observe your interaction with our prototype. I will provide you with the instructions on how to share a screen in Zoom at the session, should you have any questions.

If you are willing to participate in this study, please reply to this email with a good day and time for you to meet with us, and I will set it up in Zoom.

Thanks,

Sofya

Option 2:

Hi [Name],

Our study is looking at how people read and navigate the web. We are looking for honest feedback so that we can make necessary improvements.

A few additional things to know: 

• You will need to join from your computer and be able to share your screen with us so that we can observe your interaction with our prototype. I will show you how to share a screen in Zoom at the session, if you have questions.

• Your responses will be anonymous, and you will not be asked to share your personal information.

• The session will be recorded for strictly internal purposes to supplement note taking; it won't be shared outside of my project team.

• As part of my ethics protocol, I will send you a consent form with a detailed explanation of how we will use the information and what the research is about. 

• There will be one other person -- my designer -- on the call, observing your experience and taking notes for me so that I can focus on our conversation.

If you are okay with all this, could you find a 30 minute window on either of the days below to have a zoom call with me? 

• Mon 5/24: 1-2:30 pm

• Tue 5/25: 11 am-2pm

• Wed 5/26: 9-11:30 am; 1-2 pm

• Thu 5/27: all day

• Fri 5/2: 9-12; 2-4

Please reply to this email to let me know what works best with your schedule, and I will set it up in Zoom.

Thank you!

Sofya

2nd email: Zoom invitation

Subject: Help us design intuitive government websites

Body:  

Hi [Name]

Thank you for agreeing to provide your feedback! Below, please find a link to join our Zoom meeting on the scheduled day.

I am looking forward to our session.

Best,

Sofya

---

Sofya Savkina is inviting you to a Zoom meeting scheduled for [Day/Time]

Join Zoom Meeting

Add zoom link

3d email: consent form

Hi [name]

As I mentioned before, here is a consent form [LINK to your consent form here] for you to fill out before our session. It contains additional details about this study so that you know what to expect. If you are unable to complete the form or have questions, we can go over it together at the beginning of the session.

Thanks very much!

6. Timeline 

The following timeline is intended for this study:

• Planning: April 26-May 7

• User recruitment: May 10-21

• Conducting sessions: Week of May 24

• Analysis: June 1-4, 15-16; 18, 21-23

• Findings share-out: June 23

• Improvement discussion: June 24

• Documenting recommendations: June 25 and 28

7. Discussion Guide

Introductions, brief explanation/preview of session and topic of discussion (2 min) 

Thanks for your time today. 

Our team is conducting research to learn how people read and navigate the internet. We developed a prototype to test our assumptions, and we need your honest feedback. 

[Introduce the team]  

Interview logistics (2 min)

We have about 35 min together today, please let me know if you need to take a break for whatever reason. 

Before we begin, is it okay if I record this session? It's only for our team's use to refer back to for analysis, and it won't be shared outside of our team. We will remove your personal information to protect your anonymity. 

Do you have any questions for me before we begin? 

Context of use (5 min) 

Let's start with a couple of general questions: 

• Tell me what made you laugh recently? (Icebreaker)

• What is the government website that you used last time? When was it? How was your experience? What was hard? What was easy?

• • Can you show me?
• • How do you know it's an official government website?

• Ask if applicable and time permits

[Start screen share]  

Testing (15 min) 

[Ensure screen is being shared]  

Instructions: 

I will give you a scenario and three tasks to complete, one at a time.

Please know that this is NOT a test of you or your skills whatsoever. Our goal is to learn what works and what doesn't work for you. Please let me know if something doesn't make sense. We are here to improve things, and that's how we learn. Do you have any questions? 

[Share a prototype link with the participant] 

[When guiding - for participant the bar appears at the top and the chat is in the ... menu]

Scenario: 

Imagine that you need to understand what bullying is and the magnitude of the issue of bullying in the United States. You start researching the topic of bullying on the internet and come across this page.

[Share a prototype link with the participant] 

Task 1: Take a minute or two to review this page. 

[Remind the participant not to click ahead]

[wait for 1-2 min, in silence. As they read, watch at which point they would want to click on the link] 

• Tell me what is it that you see? 

• What do you think you can do here?

• What resources are available to you here?

• How do you know it's an official government website?

Task 2: Take a look at the "Definition of bullying" section.

• What can you do here?

• Why do you think the text is blue and underlined?

Task 3: Find a section that provides data on bullying. 

• What do you think you can do here?

• Where do you think this will take you?

• or how do you expect [this link] to work? 

• Click on this link for me. 

• Is this what you expected to be taken to?

[Avoid using words "link", "external", "icon". Only use the word "link" after the participant says the word "Link"]

Post-test questions (3 min) 

• How would you describe your experience with this information? 

• Is there anything else that you want me to know that we haven't discussed today?

• Do you have any questions for me?

• Do my observers have any questions? 

Thank you! 

8. Resources

Prototype links:

• version A [desktop]

• version A [mobile]

• version B [desktop]

• version B [mobile]