Skip to content

Latest commit

 

History

History
242 lines (182 loc) · 8.51 KB

README.md

File metadata and controls

242 lines (182 loc) · 8.51 KB

Implementation Report

This directory contains the WoT Thing Description Implementation report and the sources for it. The report itself is in report.html. Do not edit this file, it is autogenerated by executing npm run assertions from the main directory, and any edits would be overwritten the next time you generate it.

Adding Your Results

If you want to update your input for a Member organization, please go to WoT IG testing and see the README there.

Updating Implementation Report

w3c/wot-testing Repository: Updating Results from Inputs

Test inputs are collected via the WoT IG and are currently in wot/testing/tests/2019-05.


First Time Only

To update a test run to incorporate new results or test cases, first check out the wot repo,

git clone https://github.com/w3c/wot-testing.git

and install the dependencies for the AssertionTester tool:

git clone https://github.com/thingweb/thingweb-playground.git
npm install

Go to the wot/testing/tests/2019-05 directory. It contains a subdirectory for descriptions and one for inputs, which in turn contains TDs (TDs of the same implementation must be grouped in a further subdirectory) and manual assertion results as CSV files. Each is organized by Member organization.

To run all tests, while in the wot/testing/tests/2019-05 directory, run the following:

./update.sh | tee update.log

This puts all the results in the outputs directory and also logs diagnostic output to update.log. To check for errors do

grep ERROR update.log > errors.log
grep INVALID update.log >> errors.log

If there are no fatal errors (for example, a TD that does not validate or is not JSON), this file should be empty.

The output data files are now in outputs/*/*.csv. Check that there are no zero-length files in this list using ls -l. If there are it means something has gone wrong, most likely an input CSV file with a syntax error. An easy way to check for CSV syntax errors is to open the file in github. If it does not render properly as a table, there is an error, and GitHub will even point to the offending line.

w3c/wot-thing-description: Importing Results

Go back to the wot-thing-description/testing directory and update the inputs for the report generator in the inputs/implementations subdirectory:

cd ./wot-thing-description/testing/inputs/implementations
rm *.html
cp ../../../../wot/testing/tests/2019-05/descriptions/*/*.html .

And in the inputs/results subdirectory:

cd ../results
rm *.csv
cp ../../../../wot/testing/tests/2019-05/outputs/*/*.csv .

w3c/wot-thing-description: Generating Report

Now return to the main wot-thing-description directory and regenerate the report:

cd ../../../
npm run assertions

If you need to update the at-risk highlighting in the TD specification, you may also need to run npm run render afterwards. If new assertions have been added to the TD specification then you will need to run npm run render BEFORE generating the report.

Suppressing Assertions

The file suppressed.csv can be used to list assertions for which test results should be ignored. Such results will also not be used for "roll-up" results (when a child assertion, one with an underscore in its name, is used to break down assertions with multiple options into simpler, separately testable assertions).

Manual Assertions

As already mentioned, manual.csv can be used to identify assertions that need manual testing or declarations.

At-Risk Highlighting

The file atrisk.css provides highlighting for at-risk elements in the report but should NOT be edited directly as it is autogenerated. Instead edit inputs/atrisk.csv.

Report Template

The HTML template for the Implementation Report is in inputs/template.html. If you wish to edit the main explanatory text of the report or update the metadata (date, authors, etc.), do so here.

Implementation Descriptions

Implementations are described in HTML files in inputs/implementations, with one file per contributing organization. The IDs declared in these descriptions should be unique and descriptive as they will be used elsewhere, for example, in the interoperability data files referring to those implementations. To add a new description, use the template in inputs/implementations/template.html.t.

Implementations should also be entered into the table in inputs/impl.csv with identifiers, titles, and roles consistent with those assigned in the above HTML files. An implementation can either be a consumer, a producer (eg exposing a TD, a Thing), or both (eg a Servient).

Test Specifications

A procedure for testing each normative assertion should be given in inputs/testspec.html. These may be included as an appendix in the report, but are currently suppressed.

Test specifications can be given both for assertions given in the specification (see inputs/results/template.csv.t for an automatically generated list of identifiers) and for any extra assertions in inputs/extra-asserts.html.

Extra Assertions

Assertions used for testing but not (yet) included in the specification may be listed in inputs/extra-asserts.html. The intention is that these assertions should eventually, and before final release, be inserted into the final specification.

At-Risk Assertions

Assertions related to features that are at risk of being deleted from the final CR should be identified in the inputs/atrisk.csv file. The assertion text for these will be given a special color in the report table.

Categories

Assertions can be assigned to a category in inputs/categories.csv. Eventually the table may be sorted to group categories into sections; for now it is just an extra column.

Dependencies

Dependencies between assertions can be recorded in inputs/depends.csv.
The "Parents" column relates detailed assertions to more general assertions. the "Contexts" column indicates assertions that only need to be considered in a particular context (either syntactic, if pointing at another vocabulary item, or logical, if pointing at another optional assertion). Both "Parents" and "Contexts" may have multiple items separated by spaces. Entries should be IDs of other assertions. Use "null" if there is no dependency.

Result Data

Each implementation should record which features they have implemented and tested under the inputs/results directory. All data will be read and merged into the report. Mark each implemented feature with a status of either "pass" (if it satisfies the specification) or "fail" (if it does not).
If you have not implemented a feature, list its status as "not-impl".

Features not listed will not be included in the sums; this is distinct from "not-impl" as absence of a feature is meant to be used to allow different features to be reported in different files.

If you did not implement a feature on purpose please indicate this explicitly.

Any other status will be ignored (e.g., "null" as used in the template). If you have tested a feature in multiple implementations check in one file per implementation, using as a filename the id given in the template for the implementations' description. Use a convention like ORG-IMPL-MODULE.csv for the filename. The filename should also be used as an id in the description of each implementation.

The inputs/results/template.csv file lists all features but with a "null" status. Do not edit this file; it is autogenerated. It is provided so you can use it as a reference and as a basis for your own data files.

Files should be in CSV format, including headers as defined in inputs/results/template.csv, and will be parsed by the csvtojson Node.js library.

To Dos

  • Test specs need to be completed.
  • Interop and results data needs to be collected and collated.
  • Sort categories together in the output and put the category in a section header instead of a column.
  • Sanity check that parent assertions are only marked as "pass" if all sub-assertions are also marked as "pass".