Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Updates to TESTING.asciidoc for REST testing #52862

Merged
merged 1 commit into from
Feb 27, 2020
Merged

Updates to TESTING.asciidoc for REST testing #52862

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Jump to
Jump to file
Failed to load files.
Diff view
Diff view
60 changes: 42 additions & 18 deletions TESTING.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -270,34 +270,49 @@ code 137 (128 + 9, where 9 means SIGKILL).

== Testing the REST layer

The available integration tests make use of the java API to communicate with
the elasticsearch nodes, using the internal binary transport (port 9300 by
default).
The REST layer is tested through specific tests that are shared between all
the elasticsearch official clients and consist of YAML files that describe the
The REST layer is tested through specific tests that are executed against
a cluster that is configured and initialized via Gradle. The tests
themselves can be written in either Java or with a YAML based DSL.

YAML based REST tests should be preferred since these are shared between all
the elasticsearch official clients. The YAML based tests describe the
operations to be executed and the obtained results that need to be tested.

The YAML files support various operators defined in the link:/rest-api-spec/src/main/resources/rest-api-spec/test/README.asciidoc[rest-api-spec] and adhere to the link:/rest-api-spec/README.markdown[Elasticsearch REST API JSON specification]
The YAML tests support various operators defined in the link:/rest-api-spec/src/main/resources/rest-api-spec/test/README.asciidoc[rest-api-spec] and adhere to the link:/rest-api-spec/README.markdown[Elasticsearch REST API JSON specification]
In order to run the the YAML tests, the relevant API specification needs
to be on the test classpath. Any gradle project that has support for REST
tests will get the primary API on it's class path. However, to better support
Gradle incremental builds, it is recommended to explicitly declare which
parts of the API the tests depend upon.

For example:
---------------------------------------------------------------------------
restResources {
restApi {
includeCore '_common', 'indices', 'index', 'cluster', 'nodes', 'get', 'ingest'
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is the underscore prefix to "common" required?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah it's required to match the pattern

This will result in a pattern match of _common*/**. The file we are matching is _common.json. The api's don't use use folders, so we could change the pattern match to be *common*.json, however the tests do use folders so those would need to stay <prefix>*/** (or the like). I figured a simple prefix match for both would be simplest to reason about, but am open to change (in a different PR)

}
}
---------------------------------------------------------------------------

YAML tests that include x-pack specific APIs need to explicitly declare
which APIs are required through a similar `includeXpack` configuration.

The REST tests are run automatically when executing the "./gradlew check" command. To run only the
REST tests use the following command:

---------------------------------------------------------------------------
./gradlew :distribution:archives:integ-test-zip:integTest \
-Dtests.class="org.elasticsearch.test.rest.*Yaml*IT"
./gradlew :distribution:archives:integ-test-zip:integTestRunner \
--tests "org.elasticsearch.test.rest.IntegTestZipClientYamlTestSuiteIT"
---------------------------------------------------------------------------

A specific test case can be run with
A specific test case can be run with the following command:

---------------------------------------------------------------------------
./gradlew :distribution:archives:integ-test-zip:integTest \
-Dtests.class="org.elasticsearch.test.rest.*Yaml*IT" \
-Dtests.method="test {p0=cat.shards/10_basic/Help}"
./gradlew ':distribution:archives:integ-test-zip:integTestRunner' \
--tests "org.elasticsearch.test.rest.IntegTestZipClientYamlTestSuiteIT" \
-Dtests.method="test {p0=cat.segments/10_basic/Help}"
---------------------------------------------------------------------------

`*Yaml*IT` are the executable test classes that runs all the
yaml suites available within the `rest-api-spec` folder.

The REST tests support all the options provided by the randomized runner, plus the following:

* `tests.rest[true|false]`: determines whether the REST tests need to be run (default) or not.
Expand Down Expand Up @@ -459,6 +474,14 @@ version 5.3.2 run:
./gradlew v5.3.2#bwcTest
-------------------------------------------------

Use -Dtest.class and -Dtests.method to run a specific bwcTest test.
For example to run a specific tests from the x-pack rolling upgrade from 7.7.0:
-------------------------------------------------
./gradlew :x-pack:qa:rolling-upgrade:v7.7.0#bwcTest \
-Dtests.class=org.elasticsearch.upgrades.UpgradeClusterClientYamlTestSuiteIT \
-Dtests.method="test {p0=*/40_ml_datafeed_crud/*}"
-------------------------------------------------

Tests are ran for versions that are not yet released but with which the current version will be compatible with.
These are automatically checked out and built from source.
See link:./buildSrc/src/main/java/org/elasticsearch/gradle/VersionCollection.java[VersionCollection]
Expand Down Expand Up @@ -512,8 +535,9 @@ There are multiple base classes for tests:
* **`ESIntegTestCase`**: An integration test case that creates a cluster that
might have multiple nodes.
* **`ESRestTestCase`**: An integration tests that interacts with an external
cluster via the REST API. For instance, YAML tests run via sub classes of
`ESRestTestCase`.
cluster via the REST API. This is used for Java based REST tests.
* **`ESClientYamlSuiteTestCase` **: A subclass of `ESRestTestCase` used to run
YAML based REST tests.

=== Good practices

Expand Down Expand Up @@ -643,4 +667,4 @@ please see https://esrally.readthedocs.io/en/stable/[Rally's documentation].

The Elasticsearch docs are in AsciiDoc format. You can test and build the docs
locally using the Elasticsearch documentation build process. See
https://github.com/elastic/docs.
https://github.com/elastic/docs.