autotest

Accessibility test automation

Summary

Autotest is a prototype application investigating methods of web-application test automation, with an emphasis on tests for accessibility. Autotest performs automated tests on web documents.

System requirements

Version 14 or later of Node.js.

Technologies

Autotest uses the dotenv package to keep secrets and personalize form completion.

Autotest uses Node.js to run:

• an HTTP(S) server to interact with you
• an HTTP(S) client to interact with websites

Autotest uses the Playwright package to launch browsers, perform user actions in them, and perform tests.

Autotest uses:

• accessibility-checker to perform the ibm test, using the IBM Equal Access Accessibility Checker
• alfa to perform the alfa test, using Siteimprove alfa
• Automated Accessibility Testing Tool to perform the aatt test, using HTML CodeSniffer
• axe-playwright to perform the axe test, using axe-core
• KickFire Company Name to Website Endpoint to convert organization names to URLs
• pixelmatch to measure motion
• WAVE API to perform the `wave' test, using WAVE

Autotest enables federated accessibility testing by combining test packages and supplementing them with additional tests. As of January 2022, the counts of tests in the packages referenced above were:

• aatt: 98
• alfa: 103
• axe: 138
• ibm 163
• wave: 110
• total: 612

Code organization

Some of the files of code are located at the root of the project directory.

The main subdirectories of the project directory containing other code files are:

• tests: files containing the code defining particular tests
• procs: shared procedures
• validation: code and artifacts for the validation of tests

Autotest assumes that you also have code in another directory. There you may name the main subdirectories as you wish, such as:

• scripts: instructions for conducting tests
• batches: collections of URLs to be tested by a single script
• reports: reports of the results of tests
• data: data for use by procedures

Installation

git clone autotest
cd autotest
npm install

Configuration

Create a file named .env in the project directory, with this content:

HOST=localhost:3000
PROTOCOL=http
WAVE_KEY=[1]
KICKFIRE_KEY=[2]
SCRIPTDIR=[3]
BATCHDIR=[4]
REPORTDIR=[5]
DATADIR=[6]

Replace the bracketed numbers above with the applicable values, such as:

SCRIPTDIR=../my-autotest/scripts

You are responsible for obtaining keys to use WAVE or Kickfire if you wish to use them within Autotest. (See the links above under “Technologies”.)

Specification

Before using Autotest, you must specify what it should do. You do this by creating at least one script, and optionally one or more batches.

Validation

In order to validate Autotest tests, you will use a validation script in the validation/scripts directory, plus artifacts (HTML documents) in a subdirectory of the validation/targets directory. For any validation script named foo.json, there must be a subdirectory named validation/targets/foo. For several of the tests in Autotest, validation scripts and targets already exist, and you can use them as-is. They are named after the tests that they validate.

Scripts

Introduction

Autotest, whether you test with it or validate its tests, performs the commands in a script. When you run Autotest, you first tell it which mode you want (running or validating). Then it lists the available scripts for the mode you chose and asks you which script you want it to execute.

A script is a JSON file with the properties:

{
  "what": "string: description of the script",
  "strict": "boolean: whether redirections should be treated as failures",
  "commands": "array of objects: the commands to be performed"
}

Example

Here is an example of a script:

{
  "what": "Test example.com with alfa",
  "strict": true,
  "commands": [
    {
      "type": "launch",
      "which": "chromium",
      "what": "Chromium browser"
    },
    {
      "type": "url",
      "which": "https://example.com/",
      "what": "page with a few accessibility defects"
    },
    {
      "type": "test",
      "which": "alfa",
      "what": "Siteimprove alfa package"
    }
  ]
}

This script tells Autotest to open a page in the Chromium browser, navigate to example.com, and perform the tests in the alfa package on that URL. Autotest will output a report in your browser window and save a copy of the report as a file.

Strictness

If the strict property is true, Autotest will accept redirections that add or subtract a final slash, but otherwise will treat redirections as failures.

Commands

Commands in general

The commands property’s value is an array of commands, each formatted as an object.

Each command has a type property and optionally has a name property. It must or may have other properties, depending on the value of type.

You can look up the command types and their required and optional properties in the commands.js file. That file defines two objects, named etc and tests.

The properties of the etc object specify what is required for commands to be valid. Here is one example of a command:

{
  "type": "link",
  "which": "warming",
  "what": "article on climate change"
}

And here is the applicable property of the etc object:

link: [
  'Click a link (which: substring of its text; what: description)',
  {
    which: [true, 'string', 'hasLength'],
    what: [false, 'string', 'hasLength']
  }
]

As this example illustrates, to find the requirements for a link-type command, you look up the link property of the etc object. Its value is an array with two elements: a string describing the command and an object containing requirements.

A requirement, such as which: [true, 'string', 'hasLength'], has a property key as its key. In this case, it specifies what is required for the which property of a link-type command. The requirement takes the form of an array:

• 0. Is the property (here which) required (true or false)?
• 1. What format must the property value have ('string', 'array', 'boolean', or 'number')?
• 2. What other validity criterion applies (if any)? (Omitted if none.)

That other validity criterion may be any of these:

• 'hasLength': is not a blank string
• 'isURL': is a string starting with http, https, or file, then ://, then ending with 1 or more non-whitespace characters
• 'isBrowserType': is 'chromium', 'firefox', or 'webkit'
• 'isFocusable': is 'a', 'button', 'input', 'select', or 'option'
• 'isState': is 'loaded' or 'idle'
• 'isTest': is the name of a test
• 'isWaitable': is 'url', 'title', or 'body'
• 'areStrings': is an array of strings

Test commands

Commands of type test may have additional validity requirements. They must always conform to the test object in etc, namely:

test: [
  'Perform a test (which: test name; what: description)',
  {
    which: [true, 'string', 'isTest'],
    what: [false, 'string', 'hasLength']
  }
]

Thus, each test-type command must have a which property and may have a what property. (As stated above, it may also have a name property.)

The commands.js file also has a tests object. Its properties provide additional validity requirements for some tests. For example, one property of tests is:

motion: [
  'Perform a motion test (delay: ms until start; interval: ms between; count: of screenshots)',
  {
    delay: [true, 'number'],
    interval: [true, 'number'],
    count: [true, 'number']
  }
]

This property says that any motion-type test must, in addition to the required and optional properties of all tests, also have delay, interval, and count properties with number values.

The meanings of the extra properties of test commands are stated, in parentheses, in the first element of the array, except for two properties:

• withItems. The true or false value of withItems, in any test requiring it, specifies whether the report of the results of the test should itemize the successes and failures.
• expect. Any command that will validate a test must contain an expect property. The value of that property states an array of expected results. If an expect property is present in a test command, the report will tabulate and identify the expectations that are fulfilled or are violated by the results. For example, a test command might have this expect property:

"expect": [
  ["total.links", "=", 5],
  ["total.links.underlined", "=", 2],
  ["total.links.outlined"]
]

That would state the expectation that the result property of the report will have a total.links property with the value 5, a total.links.underlined property with the value 2, and no total.links.outlined property.

Command sequence

The first two commands in any script have the types launch and url, respectively, as shown in the example above. They launch a browser and then use it to visit a URL. For example:

{
  "type": "launch",
  "which": "chromium",
  "what": "Open a page in a Chromium browser"
}

{
  "type": "url",
  "which": "https://en.wikipedia.org/wiki/Main_Page",
  "what": "English Wikipedia home page"
}

Command types

The subsequent commands can tell Autotest to perform any of:

• moves (clicks, text inputs, hovers, etc.)
• navigations (browser launches, visits to URLs, waits for page conditions, etc.)
• alterations (changes to the page)
• tests (whether in dependency packages or defined within Autotest)
• scoring (aggregating test results into total scores)
• branching (continuing from a command other than the next one)

An example of a move is:

{
  "type": "radio",
  "which": "No",
  "index": 2,
  "what": "No, I am not a smoker"
}

In this case, Autotest checks the third radio button whose text includes the string “No” (case-insensitively).

In identifying the target element for a move, Autotest matches the which property with the texts of the elements of the applicable type (such as radio buttons). It defines the text of an input element as the concatenated texts of its implicit label or explicit labels, if any, plus, for the first input in a fieldset element, the text content of the legend element of that fieldset element. For any other element, Autotest defines the text as the text content of the element.

When multiple elements of the same type have indistinguishable texts, you can include an index property to specify the index of the target element, among all those that will match.

An example of a navigation is the command of type url above. Another is:

{
  "type": "wait",
  "which": "travel",
  "what": "title"
}

In this case, Autotest waits until the page title contains the string “travel” (case-insensitively).

An example of an alteration is:

{
  "type": "reveal",
  "what": "make everything visible"
}

This command causes Autotest to alter the display and visibility style properties of all elements, where necessary, so those properties do not make any element invisible.

An example of a packaged test is:

{
  "type": "test",
  "which": "wave",
  "reportType": 1,
  "what": "WAVE summary"
}

In this case, Autosest runs the WAVE test with report type 1.

An example of an Autotest-defined test is:

{
  "type": "test",
  "which": "motion",
  "delay": 1500,
  "interval": 2000,
  "count": 5,
  "what": "test for motion on the page"
}

In this case, Autotest runs the motion test with the specified parameters.

Near the top of the index.js file is an object named tests. It describes all available tests.

An example of a scoring command is:

{
  "type": "score",
  "which": "asp09",
  "what": "5 packages and 16 custom tests, with duplication discounts"
}

In this case, Autotest executes the procedure specified in the a11y05 score proc (in the procs/score directory) to compute a total score for the script. The proc is a JavaScript module whose scorer function returns an object containing a total score and the itemized scores that yield the total.

The scorer function inspects the script report to find the required data, applies specific weights and formulas to yield the itemized scores, and combines the itemized scores to yield the total score.

The data for scores can include not only test results, but also log statistics. Autotest includes in each report the properties:

• logCount: how many log items the browser generated
• logSize: how large the log items were in the aggregate, in characters
• prohibitedCount: how many log items contain (case-insensitively) 403 and status, or prohibited
• visitTimeoutCount: how many times an attempt to visit a URL timed out
• visitRejectionCount: how many times a URL visit got an HTTP status other than 200 or 304

An example of a branching command is:

{
  "type": "next",
  "if": ["totals.invalid", ">", 0],
  "jump": -4,
  "what": "redo search if any invalid elements"
}

This command checks the result of the previous act to determine whether its result.totals.invalid property has a positive value. If so, it changes the next command to be performed, specifying the command 4 commands before this one.

A next-type command can use a next property instead of a jump property. The value of the next property is a command name. It tells Autotest to continue performing commands starting with the command having that value as the value of its name property.

URL commands

Once you have included a url command in a script, you do not need to add more url commands unless you want the browser to visit a different URL.

However, some tests modify web pages. In those cases, Autotest performs additional url commands after those tests, to ensure that changes made by one test do not affect subsequent acts. Those additional url acts appear in the report, but their commands are not inserted into your script.

Batches

There are two ways to use a script in run mode to give instructions to Autotest:

• The script can be the complete specification of the job.
• The script can specify the operations to perform, and a batch can specify which pages to perform them on.

A batch is a JSON file with this format:

{
  "what": "Accessibility standards",
  "hosts": [
    {
      "which": "https://www.w3.org/TR/WCAG21/",
      "what": "WCAG 2.1"
    },
    {
      "which": "https://www.w3.org/WAI/standards-guidelines/aria/",
      "what": "W3C WAI-ARIA"
    }
  ]
}

When you combine a script with a batch, Autotest performs the script, replacing the which and what properties of all url commands with the values in the first object in the hosts array, then again with the values in the second object, and so on. Those replacements also occur in the inserted extra url commands mentioned above.

A batch offers an efficient way to perform a uniform set of commands on every host in a set of hosts. In this way you can run the same set of tests on multiple web pages. Autotest writes a separate report file for each host.

A no-batch script offers a way to carry out a complex operation, which can include navigating from one host to another, which is not possible with a batch. Any url commands are performed as-is, without changing their URLs.

Execution

Open a command shell (a terminal), navigate to the Autotest directory, and enter node index. That starts Autotest’s server, which listens for requests on port 3000 of localhost.

To make requests to Autotest, visit localhost:3000/autotest with a web browser and follow the instructions displayed by Autotest.

Autotest outputs a report to your browser window, or a progress report if you are using a batch with two or more hosts. It also writes files:

• a report file for the script, or one report file per URL if there was a batch
• image files if necessary for exhibits in reports (i.e. if the images cannot be linked to)
• image files from screenshots made in the motion test
• temporary files required by some tests

When you have finished using Autotest, you stop it by entering CTRL-c.

Contribution

You can define additional Autotest commands and functionality. Contributions are welcome.

Accessibility principles

Autotest allows other tests to be defined, which can detect accessibility properties not discovered by any of the packages.

The rationales motivating tests can be found in comments within the files of those tests, in the tests directory.

Future work

Further development is contemplated, is taking place, or is welcomed, on:

• links with href="#"
• links and buttons styled non-distinguishably
• first focused element not first focusable element in DOM
• never-visible skip links
• buttons with no text content
• modal dialogs
• autocomplete attributes

Additional test packages may be integratable into Autotest. The bbc-a11y package has been considered, but it has not been updated since 2018 and has vulnerable dependencies.

Testing challenges

Activation

Testing to determine what happens when a control or link is activated is straightforward, except in the context of a comprehensive set of tests of a single page. There, activating a control or link can change the page or navigate away from it, interfering with the remaining planned tests of the page.

The Playwright “Receives Events” actionability check does not check whether an event is dispatched on an element. It checks only whether a click on the location of the element makes the element the target of that click, rather than some other element occupying the same location.

Test-package duplication

Test packages sometimes do redundant testing, in that two or more packages test for the same issues. But the testing and reporting are not necessarily identical.

To compensate for test-package duplication, scoring procs can adjust the weights they apply to particular findings of test packages.

Repository exclusions

The files in the temp directory are presumed ephemeral and are not tracked by git.

More information

The doc directory may contain additional information.