Component reorg treejack test results

In November, 2016 we ran two treejack tests to validate a proposed reorganization of the USWDS directory structure. (The "I" in this document is Shawn Allen.)

Test trees

The first test used the originally proposed structure, which kept the top-level src tree and reorganized its contents into abstract, assets, and components subdirectories.

The second test eliminated the src directory and moved components to the tree root, moved fonts and images from src/assets to the root, and renamed abstract to settings. The goal here was to improve the ease of completing tasks compared to the time it took users in the first test.

Observations

• Unfortunately, we only dramatically improved people's "scores" on a single task; the average score for all tasks actually dropped slightly from 4.5, which is pretty bad to begin with.

• I went into the first test armed with some fancy Sass guidelines, and didn't even consider that people wouldn't know what to make of the abstract directory. (Spoiler alert: they didn't!) Six participants responded to the question of what they would change about the file structure with a response along the lines of, "What the heck does 'abstract' mean?" In hindsight, this might have been the most obtuse name possible for a project aimed at users with little to no knowledge of Sass, let alone any specific set of wonky guidelines.

  What I found even more interesting, though, was that the directory named settings performed slightly worse than abstract. My only hope is that we can blame this on the fact that there were more participants in the second test (115 started, 63 completed) than the first (48 started, 23 completed), and likely not much overlap in the participants of both tests.

• I categorized some of our components into more specific directories, with form templates in a new patterns directory alongside components, assuming that users would understand the distinction (something like: "pattern is to component as template is to function"). Just to make things even more confusing, we decided to add an examples directory at the last minute. Users were almost universally confused by these three directories at one point or another, which I guess proves that the "component" vs. "pattern" distinction isn't as universal as I'd assumed. To quote one of the participants when asked what they would change:

  Clarify the difference between patterns, components, and settings. Condense, rename, or remove one of them.

My main takeaway is this: We can't assume anything about our users. As is often the case in design, the solution to the problem of helping people find things in our file structure appears to be relentless simplification. My hunch is that "flattening" the directory structure and removing some of the directories that exist purely for categorization (e.g. patterns vs. components, as opposed to just putting everything in components) will help people avoid going up and down the tree in search of specific things.

If you have any observations or thoughts on either our methodology or the results, please let us know in this issue.

Takeaways

• People appear to find things more easily when they are on display right at the root level.
• Conversely, the more we categorized components by nesting them in directories with similar types of things, the harder they were to find.
• We should consider taking a cue from Material Design Lite's decidedly flat directory structure, in which the src directory contains a single directory for each component, rather than attempting to categorize them in any way.

In other words: don't assume that people know the difference between "components", "patterns", or other abstract names for things.

Scoreboard

Here is a comparison of the task success rates in the first and second tests:

Where would you find/change/learn about...	First	Second	Change
available typeface options	5	8	+3
The only task that we dramatically improved was helping people find the typeface options. It looks like this a lot to do with removing the src directory nesting and placing fonts at the root, but I also have a hunch that the assets directory name is loaded for some users.
color settings to match your products branding	4	3	-1
People had a slightly harder time finding the color settings in the second test than the first. This genuinely surprised me because I specifically tried to address the poor performance of the abstract directory by renaming it to settings
code for the login or submit UI element	6	3	-3
The overall score of the date input "behavior" (in retrospect, a poor choice of wording) dropped by half from the first test to the second. As with the accordion test, the problem here appears to be the introduction of the patterns directory, which is where I thought the various form configurations should live because they're not technically components.
code that controls the display of accordions	9	7	-2
The accordion test did **really well** the first time, and slightly poorer the second. The confounding factor here appears to be the introduction of examples and patterns directories, and some confusion around what the accordion is, exactly—or maybe, more generally, what distinguishes a "component" from a "pattern".
code that controls the behavior of the date input form	6	3	-3
The overall score of the date input "behavior" (in retrospect, a poor choice of wording) dropped by half from the first test to the second. As with the accordion test, the problem here appears to be the introduction of the patterns directory, which is where I thought the form guidance should belong because it's not technically a component. This was the task that took participants the longest: between 10 and 30 seconds, nearly 21 on average.
display settings for general look and feel	1	—	—
Participants went all over the map on this one. Maybe the question was too vaguely worded, or maybe our file structure just doesn't answer it well; either way, it's a serious problem.
which parts of USWDS can be customized	—	2	—
Participants were similarly confounded by this question, and veered off into almost every path searching for the answer. Interestingly, nearly half the participants (17 of 39) descended into settings and then backed out after seeing what was in there.
files that show you how to use USWDS on your project	—	5	—
There's not a lot to point out here other than that we assumed examples would be an dead giveaway, but nearly 20% of participants went straight to the README.md rather than bothering to look anywhere else.

What didn't work

For me, the hardest thing to grapple with is a lack of data for the existing directory structure. Without this, it's hard to really know whether these new trees are any better than what we have now. If I were to do this again, I would run the first test with the existing structure, and recruit a set of participants willing to test multiple trees (preferably, in a random order).

The other thing that we should've been more careful about was rewriting the "General display settings" task in the second test. The first tree tested abysmally (see: "abstract" above), but we didn't validate the same question in the second; instead, we repurposed that task to see whether users were able to figure out which parts could be "customized" (same thing in theory, but completely different language), then added a new task to determine whether people could find the usage instructions. The repurposed task tested only barely better (a score of 2 overall, as opposed to 1), but the usage instructions tested reasonably well at 5 overall.