August Listening Tour Findings
Brad Nunnally edited this page May 9, 2017
·
9 revisions
In August of 2016, we performed a “listening tour” with teams that are actively using (or hoping to use) the Standards. The 18F team met with product teams at seven different federal agencies; with each, they spent two hours observing how teams use the USWDS and discussing those teams’ pain points with and hopes for the Standards.
What follows is a collection of themes that came out of the research. Each is accompanied by data points explaining why the theme exists, along with strategies the USWDS product team should consider undertaking to better support its users.
After we’d completed our interviews and synthesis, we on the 18F research team drew the high-level conclusions described in the following paragraphs. If you have time, we do encourage you to read the entirety of this document; that said, we recognize that time is scarce, and this summary provides a good glimpse into what we uncovered.
Barriers to contribution exist: One theme we repeatedly heard addresses contributing to the Standards. Many users we talked to felt that they encounter barriers to contribution, which can take one of several forms. Some users simply don’t have the time to contribute ideas or components to the Standards. Others weren’t aware that the Standards team is seeking outside contributions. Yet others feel that their technical ability might prevent them from contributing successfully, or that the custom components they created using the Standards might be too specialized to be of use to anyone else.
The Standards support only certain types of sites: Several teams pointed out that, while the Standards do a great job of supporting certain types of sites, they do just that: they only support certain types of sites. To be more broadly useful, the Standards could support a greater range of the types of sites agencies want to build.
Users want qualitative and quantitative proof: Nearly every team we spoke to expressed the desire for qualitative and quantitative proof of the Standards’ effectiveness — “selling points” that they could share with their teammates, senior management, or both. Different teams requested different types of proof; some recommended a portfolio or case-study gallery, while others indicated they might prefer success metrics they could quickly share with stakeholders. A few teams noted that demo sites would also be an effective way of illustrating the Standards’ benefits.
Users perceive the Standards as individual components: Although the Standards are intended to facilitate various uses (as components; as a design system), most of the people we spoke to view them solely as components, not as a framework. Several users requested more guidance on using the Standards as a system (in addition to the guidance we provide about using individual components).
Getting started instructions don’t address the diverse development environment present in the federal government: Though most of our users reported having had a straightforward experience getting started with the Standards, several pointed out that our Getting started instructions did not fit within their unique development environment. Providing alternative sets of instructions, or broadening the current set, could make the instructions more useful to a larger group of users.
The method of setting up the Standards impacts teams’ ability to smoothly upgrade to future updates: Several teams shared that one’s development environment shouldn’t impact a team’s ability to update their local copy of the Standards. Rather, any team should be able to update to the latest version of the Standards, which may have small or non-breaking changes, by perform few or no custom actions or procedures. Updates that include large-scale change should be accompanied by custom upgrading instructions to help guide teams through the upgrade process.
An updated product identity could encourage wider use: Finally, users shared that changes to the Standards’ brand identity could encourage more users to adopt them. Many of our interviewees were confused by the inclusion of Draft in the Standards’ name (and a few asked when the Standards would be “finished”); this perception that the Standards are incomplete may serve as a barrier to adoption. Other people were confused about our team’s usage of the term Standards, which has a specific meaning in a government context.
The following section details the more specific findings from which we created our major findings. These address the entire experience of learning about and using the Standards, including how users find implementation information, how users perceive our contribution guidelines, features users would like to see added, and more.
- Several people expressed confusion about how much they can deviate from the Standards. There were a lot of questions around when it is appropriate to deviate from the Standards, whether it be for choosing a darker color or designing a component that doesn’t exist yet. This concern also applies to some of the team dynamics seen in teams utilizing the Standards. If, for example, a designer is using the Standards to mock something up but breaks pixel-perfection, the engineers who are implementing the design may believe the designer was trying to customize something when really it was just a misunderstanding. This can hurt code reusability and defeat the purpose of using the Standards in the first place.
- Folks also noted wanting to learn more about how to create effective designs using the Standards as they’re currently presented. Many of the consumers of the Standards are on engineering-heavy teams that lack any permanent designer and are often starved for resources. While a designer may be able to look at a type hierarchy and know when to use an H2 versus an H4, someone without a design background may have trouble making such a distinction.
- People requested additional guidance on how to use the hierarchy-based components, such as typography.
- Numerous people asked about the meaning of the word standards in the context of our name.
- People using the HTML and CSS had different approaches. Depending on the maturity of each team’s front-end development process, usage of the Standards (and the associated stylesheets) included including the Standards as a third-party library with updates pulled directly from NPM, and opening up Chrome dev tools and copying the styles to the one element they were choosing to implement, among other uses.
- Provide more examples of other sites utilizing the Standards, along with ready-to-use HTML page templates, which could help users with fewer design resources tackle some of their page layout concerns.
- Communicate more clearly about proposed components, which will help users decide whether they need to deviate from the Standards (build a custom component) or wait a few weeks for the piece they need to be released.
- Showcase examples of how to effectively include the Standards in a project with its own custom styles — this will set users up for success when it comes to implementing updates.
- If the user need exists, showcase sample CSS alongside sample HTML for users who only want to utilize a single component from the Standards. (We could also address this situation by more formally spelling out the “right” way to use the Standards stylesheet.)
- The teams we spoke to did a good amount of customization when building more advanced components. Users who are creating complex web applications find that they still have to build quite a few custom components. While the Standards may have everything necessary to create a great documentation site, they still lack a lot of things that an application may require (including toggles, mapping, displays of numbers, complex tables, and more).
- Our components don't support customization very well.
- A few teams noted that our use of the terms Draft, Alpha, and Beta made them hesitate before using the Standards, or conveyed the idea that it would be hard to get support. Our use of the word Draft could cause friction for teams who are trying to sell the idea of Standards usage to upper management, as it implies that the Standards aren't ready to go. Alpha and Beta in the maturity scale may be perceived by dev teams as components that are too buggy to use rather than those that are lacking in feature-richness.
- Many teams wondered how updates will impact different teams' usage of the Standards. Teams that are not experienced with using third-party libraries in their projects may implement the Standards in ways that make it very difficult for updates to get pulled into the code. Copying and pasting are very common, and these can also introduce difficulties with updating.
- Examine the use of the word “Draft” if we're looking for increased adoption of the Standards. (We can also take a number of other steps to sell the Standards — these are detailed below.)
- Reevaluate our language and label choices for the component maturity scale. Currently, our wording seems open to misinterpretation.
- Communicate more effectively around when updates are coming out and what those updates include. This should help ensure users are as up-to-date as they need to be.
- Provide guidance on how to effectively include third-party stylesheets in various projects. This will help users who are copying and pasting sections of the Standards directly into their stylesheets.
- Some agencies and organizations don’t have the benefit of having a team of designers and front-end developers to implement many of the features present in the Standards.
- Teams' reliance on both internal IT departments and external vendors can make shipping updates to the front end challenging and expensive.
- Many teams aren't able to choose what technology stack they use — it's decided for them. Consequently, they're constrained by the technology made available to them.
- Provide focused content on how to use the Standards when working in one of the typical agency environments.
- Offer guides that speak to the typical development stacks that exist at federal agencies and provide insight into how the Standards integrate into them.
- Some teams using the Draft U.S. Web design Standards are unclear on how to begin using the code base and how best to deviate from components present in the code base.
- Other teams have been using frameworks like Bootstrap and Foundation and need to understand how best to merge the Standards into those frameworks.
- Teams without front-end designers may be unaware of how to customize components with custom code or styling.
- A handful of teams have had to extend or replace components to meet product specific needs (one example is data grids).
- One team has created a UI Kit, which produces web components, to allow teams to easily churn out components.
- Write tutorials on customizing and integrating the Standards into some of the major frameworks on the market.
- Investigate how the Standards can better support web components. This will make it easier for teams to implement the Standards, and it will also encourage reusability.
- Look to other frameworks, such as Bootstrap and Material, to draw inspiration on how to handle customization and guidance.
- Some teams looking to use the Draft U.S. Web Design Standards are building more complicated web applications, which the current component set doesn’t properly support. These teams are looking to the Standards to provide usage guidance on interactive elements and more complicated components, which we currently don’t offer.
- The components we currently offer are comparatively basic and are useful for teams who are looking to get started on a project. However,some teams need to better understand how to how to extend the existing components.
- Teams are looking to understand how different components can be used together in the form of an overall framework — for example, how to use search results with pagination or profile pages that focus on upper management teams.
- Some agencies have a strong visual brand that products need to adhere to. Teams from these agencies seek guidance from the Standards on how they could apply their brand while using the Standards.
- Develop a component roadmap that details when new components and guidance will be released.
- Provide guidance and (potentially) new extended components to support web application use.
- Offer tools that allow teams to easily customize the visual layer of the Standards to match their agencies’ brands.
- Provide templates that show the components working in the form of a framework (for example, Search Results).
One theme we heard from several interviewees is that our contribution policies aren’t as accessible as they could be. People we talked to noted the following:
- The fact that our contribution policies live on GitHub gives them a “hidden” quality. Many users first visit the Standards website (not one of our repos), and we don’t list the full guidelines on the site. Additionally, both the designer and developer Getting started pages feature contribution guidelines as the last section, thereby de-emphasizing them.
- Several interviewees expressed confusion about whether they’re “allowed” to contribute back to the Standards.
- Some people were confused about the types of contributions they can make. For instance, a few people were unsure about whether they could contribute findings from their user research.
- Along similar lines, several interviewees felt that contributions they’d like to make might be “too specific” (specialized) to be of value to the Standards — in effect, they felt that components that work for their agency might not be more widely applicable.
- Consider including the full contribution guidelines on the Standards site (though this might render the site content a bit unwieldy). That said, we could provide more context in each of the Contribution Guidelines sections (designers and developers), including the types of contributions we’re looking for, encouragement for submitting things that may be too specific, and more.
- If the team decides to create an external-facing WDS intro course (webinar), we should definitely include a section on contributing to the Standards. In it, we could stress that the Standards rely on outside contributions, and we could provide concrete, actionable steps folks can take to start the contribution process.
- Update the contribution guidelines to include a list of the types of contributions we accept — not just components or code, but ideas, user research, and more.
- More actively solicit contributions. Many of our interviewees mentioned that they get most of their Standards-related information on Twitter; we could periodically tweet requests for new components or ideas to see whether we can increase contribution rates.
- Finally, our documentation could emphasize that component (or idea) specificity shouldn’t be a deterrent to contribution! Encouraging folks to share any and all contributions, regardless of how tailored they are to a certain agency’s mission, might improve our contribution rates.
- Teams using the Standards often weren’t sure how to contribute back to the project. This included not being sure what mechanisms exist to allow for external contributions, as well as feeling like they (the teams) didn’t have the ability and knowledge to contribute.
- Many teams didn’t know what features they’s pull from a single component from their code case or design work to submit it as a contribution.
- Many teams using the Standards are highly focused on working on their assigned product and getting the next release out the door. These teams may have little impulse or drive to contribute back, especially in the absence of strong calls to action soliciting outside contributions.
- Relatedly, the time required to make contributions is a big constraint for some teams, which impacts their ability to contribute back to the Standards.
- What, exactly, contributing to the Standards looks like might be unclear to some users. For example, people might not understand whether they should contribute code, designs, or research.
- The process of submitting a contribution through a pull request is too high of a burden for some teams, especially those with little experience with GitHub.
- Allow for different tiers of contributions, including screenshots, full code samples, written ideas, and so on.
- Feature a contribution workflow that isn’t solely dependent on using GitHub (at least directly), and that’s more proactive which illustrates how teams can contribute to the project.
- Provide a method for teams to submit usability test findings they have collected on their product that might pertain to the Draft U.S. Web Design Standards.
- Communicate value statements around why teams should contribute back to the project.
- Users reported wanting high-level guidance on things like maps, data presentation, legislation, media, and more. Users of the Standards are likely to need many different things that the Standards don't currently provide (and likely will not provide). Designers want to ensure that the designs they are working on match the rest of the Standards, even if they are building something custom on top of the base Standards.
- Users would prefer guidelines that don't speak to specific technology. When it comes to giving guidance on things like mapping, many agencies may have strict requirements around which map provider they can use. Therefore, more generally applicable guidance may be preferable.
- Several of our users also requested justification on why certain components aren’t included. Sometimes a component isn’t included because it’s being actively developed. Sometimes it isn’t being developed because it isn’t applicable enough to fit with the rest of the Standards. There are other reasons why a component may not be included, too.
- Two other topics users reported wanting guidance on is how different components interact with each other and how to use the Standards as a system. Some teams (especially those with minimal design resources) expressed confusion on how the Standards actually fit together. In their current state, the Standards answer the question of how a component should look, but not the question of where the component should sit on a page.
- Finally, users mentioned wanting to know more about how to use images in different contexts; this includes using background images and icons, and best practices for captioning. Currently, the Standards don't contain image guidance, so users who want to build a gallery of images, for example, or use a background image in the hero area of their page are left to figure it out on their own.
- Create page templates. Users could benefit from page templates that show how different components interact with each other. Additionally, some users mentioned wanting to see examples of other implementations to get ideas around how they could adapt the Standards for their own use.
- Provide additional (and better) communication around components that are currently in development. Very few users knew about our current product roadmap, which might be an indication that we should reconsider where it lives.
- Consider adding guidance around the usage of images, at least in a few key scenarios. Based on the feedback our users gave, this seems to be a high-priority component. We could provide additional guidance around making images accessible for scenarios where their placement is less prescribed.
- Consider providing additional guidance on non-Standards topics. Many agencies could benefit from guidance around applying the Standards to things that aren’t directly covered in the Standards. We could consider adding (to the documentation site) a separate section for bonus content on things like theming map software. Alternatively, we could consider covering these topics in a series of blog posts that could be archived for future reference.
Our team has long discussed the merits of adding a case studies section or portfolio to the Standards site, and our research bears out the validity of this idea. Nearly every team we spoke to mentioned that they’d like to see examples of how other agencies are using the Standards. Additionally, many teams mentioned wanting “selling points” — benefits of the Standards that they could share with less technical team members and stakeholders. More specifically, here’s what people requested:
- Several people mentioned wanting a demo site built using the Standards — something they could show stakeholders to demonstrate the Standards’ functionality. Folks also mentioned how much they’d like a portfolio or gallery of case studies of sites using the Standards: something that demonstrates different applications of the Standards within varied contexts.
- Quantitative selling points — success metrics for the Standards — also came up in several of our conversations. Our interviewees shared that having concrete data about how the Standards might benefit one’s team could be a major selling point for wary or less technical stakeholders.
- Somewhat related to the Standards’ name (Standards) was the suggestion that our team seek endorsement from trusted governmental groups such as GPAT, NIST, and the CIO council. Receiving endorsement from these (and other similar) groups would grant the Standards increased legitimacy, perhaps making them easier to pitch to reluctant stakeholders.
- The people we spoke to also noted that talking points or other sales collateral to present to senior management would be useful; this material could be concise and fairly non-technical in nature.
- Our long-range comms plan includes creating a case studies page for standards.usa.gov, and our research validated that this is a task we should prioritize. We could begin by highlighting a few higher-profile sites using the Standards and then gradually introduce new case studies, as time allows.
- Create a non-technical one sheet to present to senior management and other stakeholders. In addition to listing more abstract benefits of the Standards, it could include quantitative benefits.
- Approach the regulatory bodies mentioned by our interviewees to determine the best route for getting endorsement or approval of the Standards.
Nearly every group we talked to raised a point about the Standards being called the Standards. Groups’ attitudes toward our name varied: Some seemed generally in favor of it, while others had questions or reservations. Some common name-related themes included the following:
- Within the context of government, the word Standards has a fairly specific meaning, and several groups mentioned this to us. While most groups understood that our Standards are really more guidelines than edicts, some interviewees wondered whether the Standards would ever be made mandatory. Related to the government-specific usage of standards, some groups recommended we seek approval from/endorsement by trusted regulatory bodies — NIST and GPAT, for example — to legitimize our self-definition of Standards.
- Related to the previous point, some people wondered what the term standards means in relation to the Web Design Standards: Are folks expected to use the Standards as a design system, or can they pick and choose certain components? How does the classification of Standards impact customizability? Is usage of the Standards enforced (or will it be)?
- Several people expressed confusion about the inclusion of Draft in the Standards’ name; a few asked, “When will the Standards be finished?” Others shared that Draft seems at odds with Standards — how can something that’s unfinished be enforced as a best practice?
- On our site and in our other documentation, we could provide more context around how we chose the name Draft U.S. Web Design Standards, along with how we define standards (for example, not strictly in the governmental sense).
- Provide additional guidance about using the Standards, stressing customizability and the option to use all of the components or only selected ones. If we choose to create case studies or a portfolio, we could highlight a range of applications of the Standards.
- More explicitly discuss what Draft means in the context of this project, communicating that, while the Standards are always evolving, they are, in fact, complete and ready to be used; possible channels for said communication include site copy and/or a blog post about the history of the Standards’ name.
- Alternatively, consider renaming the Standards.
- One thing we heard was that some users were unable to use the Standards' code because of the CMS platforms in use by their agencies (Drupal, for example). Using the Standards on a CMS can be difficult. Depending on the theme that's in place, there may be a lot of conflicting class names and css resets that make the Standards tricky to implement.
- A good number of users said they'd like recommendations on the proper starting points for using the Standards with different CMSs (for example, Wordpress Theme Underscores). There are ultimately good ways to work with the Standards if you're using a CMS (as evidenced by the BBG and others); the Standards team could provide some instruction on best practices instead of relying on the existing knowledge of the implementation teams.
- Finally, users said they'd like guidance on how components can be incorporated into different theme engines.
- Provide CMS guidance. A huge number of government sites are driven by CMS platforms like WordPress and Drupal, and while it may not be worth maintaining an “official” Standards theme, providing instructions on how to build a theme that uses the Standards could potentially trickle down to hundreds or thousands of sites.
- Build out a barebones theme. CMS-driven sites are likely to have such a variety of content styles that one theme would not solve all the problems within a given site. It may be worth it for us to build out a barebones theme showcasing how to properly set up the Standards in a maintainable way so others can learn from it. Alternatively (or in addition to this idea), we could reach out to experts in the community who may have already done something like this.
- Using the Standards allows teams to streamline the onboarding process for new team members and vendors who are just joining a project.
- When new members join a project, teams have an easier time understanding (and explaining) how the project evolved because there is a historical record tied to the use of the Standards that explains the current state.
- Teams are able to get started more quickly because the Standards give them a pre-vetted starting point that provides a strong foundation to build on.
- The Standards allow teams to focus on more complex design problems and content because basic design patterns are covered.
- Provide cases studies of teams that are using the Standards and describe how the Standards have benefited them; this will show other teams the benefits and impacts the Standards provide.
- Create tutorials that illustrate how to quickly get set up with the Standards so teams can begin customizing and extending for their product.
- Many users wondered how to stay notified about updates and compatibility changes. Some users reporting having a difficult time keeping up with the updates that we push out. These users often read about our updates via 18F's Twitter account or on the 18F blog, but that content is mixed in with a lot of other things, making it tricky to find update-related information.
- Teams' use of the Standards impacts their ability and desire to update to the latest version. How teams use the Standards varies quite a bit: Some teams have comprehensive front-end build processes using tools like Gulp, and others are copying and pasting a few lines of code at a time. Each team's process impacts its ability to update.
- The people we talked to let us know that updates only seem desirable if they introduce new features or components. If new features aren’t being introduced, many people feel it's not worth the risk of dropping in a new stylesheet and breaking something. This is especially true for teams with messier build and development processes.
- Lastly, people reported wanting insights into our release schedule and product roadmap. The product roadmap exists on our documentation site, but almost nobody knew about it.
- Consider moving or renaming the product roadmap. Users understood what it was when they found it, but didn’t seem to associate our About our work page with a place where they'd find our product roadmap.
- Write a guide to best practices for including the Standards in your project. This could benefit users who are utilizing the Standards in a more haphazard fashion. Seeing the way that the new documentation site consumes the Standards may be a helpful example.
- Pursue our own forms of communication (as the Standards team) with our community of users as opposed to relying on the larger 18F communication channels.
- Detail updates and changes on the documentation site to provide less technical users a source of record other than GitHub (which some folks don't feel comfortable using). We could maintain this content in GitHub and dynamically pull it in to the docs site using the GitHub API. We could follow a similar process to popularize Standards-related blog posts published on the official 18F blog.
Prior to the listening tour, we’d hypothesized that the content on standards.usa.gov might be too dense, and we’d written entirely new “context of use” content to replace existing documentation. The information our interviewees shared, however, painted a different picture. Most teams we talked to said they found the documentation useful (and not too long); in some cases, people wanted additional guidance:
- Most teams we talked to felt the volume of content on standards.usa.gov is appropriate. A few teams mentioned that they’d like more guidance on how to use (or when not to use) a given component. Only one team, which has more experienced designers, felt the documentation was distractingly comprehensive.
- Accessibility was a topic that came up often. Several teams indicated a need for additional guidance on accessibility concerns for specific components.
- Some teams mentioned their desire for “quicker access” to information about implementing different components. In its current state, the Standards site presents this information following more contextual information about the components, which could create a barrier to folks who face extremely tight deadlines and also need detailed implementation guidance.
- Currently, individual component pages don’t list a “last updated” date. Our interviewees said they’d like to know when things were last updated so they can be sure they’re using the most current version of a components. Our recently completed comparative analysis revealed that several other frameworks’ sites link to their GitHub repos and feature drop-downs that allow users to load previous versions of a component — changes we might consider.
- Reevaluate our current static documentation and our new “context of use” content to determine which might better serve our users. It’s possible that we may be able to incorporate some of the next context of use content into our current site copy, or we may decide that the new copy isn’t as valuable.
- Review each component’s accessibility guidance and identify how it might be made more robust. From there, we could contact members of the Accessibility guild to see if they’d be willing to partner with us in creating guidance for components without it. Additionally, we could link to additional accessibility resources for those folks who want to do further research.
- Regarding the issue of quick implementation, we could consider placing implementation instructions higher on each component page. Alternatively, we could include a notice at the top of each component page letting users know where to find these instructions.
- Add a “last updated” date to each component page, and we could more prominently direct users to our changelog.
Right now, we communicate about updates and new releases via Twitter and by publishing release notes. While these channels reach an adequate number of users, our interviewees brought up alternative communication channels that may help us reach a broader audience:
- One observation a few teams shared is that the Standards team doesn’t currently “push out” updates about the Standards; rather, our communication style requires users to seek out this information.
- Our interviewees also noted that our team provides notifications of updates after the fact, rather than providing notice before and after an update is made. Participants showed interest in receiving notifications about soon-to-be-made updates.
- In addition to using our current communication channels, we could consider adding others that “push out” information to our users and prospective users. Some options include listserv messages, a Standards-specific newsletter, items in 18F’s external newsletter, and so on.
- If possible, we could publicize changes to the Standards before and after those changes are made; this would give users more time to plan for said changes. (Twitter would be a great medium for this task.)
- Several teams experienced confusion and pushback when stakeholders and development teams saw the Alpha and Beta labels on components. Many folks interpreted these labels to mean that the components in question weren't ready to be used.
- Some teams reported being unclear on whether they should use components that are currently listed as Alpha or Beta.
- Given the risk aversion common in government, the presence of words like Alpha and Beta made teams consider using other solutions or creating a homebrew solution.
- Our use of terms like Alpha and Beta “Shows 18F’s bias toward an engineering mindset,” which can be offputting for some organizations.
- It's hard for some teams to find the explanation behind the component maturity scale, which means the labels are shown without context of their meaning.
- Rethink the need for and purpose of the component maturity scale.
- Improve the wayfinding to any content that might speak to the maturity scale.
- Balance the “18F bias toward engineering” to prevent putting potential users off.