Skip to content

Parallelization Test Plan

Jump to bottom Edit New page
Selena edited this page Jan 15, 2018 · 46 revisions

Parallelization Test Plan

Parallelization Use Cases

This is a high level description of the parallelization use cases that need to be covered by automated tests. The test plan needs to cover all four modes of parallel execution with and without dependencies. Use cases are coarse-grained artifacts. There are numerous scenarios that need to be covered for each of the items listed below: Data providers, factories, sequential suite execution, parallel suite execution and various flavors of dependency conditions.

Use Case ID Description
PTP-UC-1 Parallel by methods without dependencies.
PTP-UC-2 Parallel by methods with dependencies.
PTP-UC-3 Parallel by tests without dependencies.
PTP-UC-4 Parallel by tests with dependencies.
PTP-UC-5 Parallel by classes without dependencies.
PTP-UC-6 Parallel by classes with dependencies.
PTP-UC-7 Parallel by instances without dependencies.
PTP-UC-8 Parallel by instances with dependencies.

Parallelization Test Matrix

This test matrix breaks down each of the parallelization use cases into fine-grained test cases. The test matrix is a comprehensive statement what must be tested. It does not address how to implement the test cases, nor does it cover what the inputs are or what the expected outcomes are.

PTP-UC-1: Parallel by methods without dependencies.

Test Case ID Suite execution mode Data Provider (Non-Parallel) Factory Dependencies
PTP-TC-1 Sequential No No N/A
PTP-TC-2 Parallel No No N/A
PTP-TC-3 Sequential Yes No N/A
PTP-TC-4 Parallel Yes No N/A
PTP-TC-5 Sequential No Yes N/A
PTP-TC-6 Parallel No Yes N/A
PTP-TC-7 Sequential Yes Yes N/A
PTP-TC-8 Parallel Yes Yes N/A

PTP-UC-2: Parallel by methods with dependencies

Test Case ID Suite execution mode Data Provider (Non-Parallel) Factory Dependencies
PTP-TC-9 Sequential No No Method, Intra-class group, Inter-class group, Intra-class meta group, Inter-class meta group, Intra-suite meta group, Inter-suite meta group
PTP-TC-10 Parallel No No Method, Intra-class group, Inter-class group, Intra-class meta group, Inter-class meta group, Intra-suite meta group, Inter-suite meta group
PTP-TC-11 Sequential Yes No Method, Intra-class group, Inter-class group, Intra-class meta group, Inter-class meta group, Intra-suite meta group, Inter-suite meta group
PTP-TC-12 Parallel Yes No Method, Intra-class group, Inter-class group, Intra-class meta group, Inter-class meta group, Intra-suite meta group, Inter-suite meta group
PTP-TC-13 Sequential Yes Yes Method, Intra-class group, Inter-class group, Intra-class meta group, Inter-class meta group, Intra-suite meta group, Inter-suite meta group
PTP-TC-14 Parallel Yes Yes Method, Intra-class group, Inter-class group, Intra-class meta group, Inter-class meta group, Intra-suite meta group, Inter-suite meta group

Testing methodology

There are some general areas of concern for a parallelization test. They must cover the following bases:

  • That parallel threads are spawned according to the parallelization mode specified.
  • That parallel threads are spawned in accordance with the thread pool limitations specified -- suite thread pool size, data provider thread pool size, and thread counts.
  • That true parallelization is actually taking place. It is possible for multiple threads to be spawned when parallelization is enabled, but they do not actually run in parallel, but rather one after another in sequential fashion.
  • When there are dependencies involved, that topological sorting is preserved so that methods execute only after groups and methods they depend on have executed and passed.

In order to successfully verify all the above, significant bookkeeping is required to accurately track the state of the model execution run:

  • The start and end times of each suite (DONE)
  • The start and end times of each defined 'test' within the suites (DONE)
  • The start and end time of each test method (DONE)
  • The thread pool ID and thread IDs for test method (PARTIAL -- Thread ID is now logged)
  • The order of execution of the test methods (DONE)
  • When there are multiple instances of test classes, the methods in the class also need to be mapped to the instance to which they belong (DONE)
  • When a new thread is spawned, the thread count is recorded. (TBD)
  • When a thread completes execution, the thread count is recorded. (TBD)

There is an execution state tracker which provides an interface to a synchronized data structure that holds all this information: test.thread.parallelization.TestNgRunStateTracker.

There is a listener class implementation for the suite, test and test method listeners interfaces that writes state information using the execution state tracker: test.thread.parallelization.TestNgRunStateListener.

Parallelization Test Cases

Simple Test cases for parallelization by methods

In the simple parallelization test case, the following conditions apply:

* There are NO configuration methods
* All test methods pass
* NO ordering is specified
* **`group-by-instances`** is NOT set
* There are no method exclusions

TEST CASES FOR SEQUENTIAL SUITES VERIFY:

  1. All methods in one suite are executed before any methods in subsequent suites are executed
  2. That suite-level and test-level events run in the same thread

TEST CASES FOR PARALLEL SUITES VERIFY:

  1. That the suites are run in parallel, but that their tests are run sequentially
  2. That the test methods in the tests for each suite are executed in parallel fashion according to their individual thread count limitations
  3. That when the number of suites executing in parallel reaches the specified suite thread pool size, that no additional suite-level threads are spawned until the number of suites executing in parallel drops below the specified suite thread pool size
  4. That when the number of suites executing in parallel drops below the suite thread pool size and there are still suites in the queue, a new suite begins execution.
  5. That suite-level and test-level events run in the same thread for each suite executing in parallel
  6. That separate threads in which suites execute are run in true parallel fashion and not in sequential order

ALL TEST CASES VERIFY:

  1. That method-level events (method listener events in addition to the execution of the body of the test method) occur in the same thread for a given method in a given test class instance
  2. That different test methods and their associated listener events are executing in different threads
  3. The separate threads in which test methods execute are run in true parallel fashion and not in sequential order
  4. When the number of methods executing in parallel for a given suite reaches the specified thread count, that no additional method-level threads are spawned until the number of methods executing for that suite drops below its specified thread count
  5. That when thread count drops below the maximum for a given suite and there are still methods in the queue for that suite, a new method begins execution

PTP-TC-1: Parallel by methods mode with sequential test suites, no dependencies and no factories or data providers.

SCENARIOS:

  1. Single suite with a single test consisting of a single test class with five methods

    • Thread count and parallel mode are specified at the suite level
    • The thread count is equal to the number of test methods, ensuring that no method should have to wait for any other method to complete execution

  2. Three suites with 1, 2 and 3 tests respectively. One test for a suite shall consist of a single test class while the rest shall consist of more than one test class.

    • For one of the suites, the thread count and parallel mode are specified at the suite level
    • For one of the suites, the thread count and parallel mode are specified at the test level
    • For one of the suites, the parallel mode is specified at the suite level, and the thread counts are specified at the test level (thread counts for each test differ)
    • The thread count is less than the number of test methods for the tests in two of the suites, so some methods will have to wait for the active thread count to drop below the maximum thread count before they can begin execution.
    • The thread count is more than the number of test methods for the tests in one of the suites, ensuring that none of the methods in that suite should have to wait for any other method to complete execution

PTP-TC-2: Parallel by methods mode with parallel test suites, no dependencies and no factories or data providers.

SCENARIOS:

  1. Two suites with 1 and 2 tests respectively. One test for a suite shall consist of a single test class while the rest shall consist of more than one test class.

    • The suite thread pool is 2, so neither suite should have to wait for the other to complete execution
    • For both suites, the thread count and parallel mode are specified at the suite level
    • The thread count is less than the number of test methods in a test for one of the suites, so some methods will have to wait for the active thread count to drop below the maximum thread count before they can begin execution.
    • The thread count is more than the number of test methods for the rest of the tests, ensuring that none of their test methods should have to wait for any other method to complete execution

  2. Three suites with 1, 2 and 3 tests respectively. One test for a suite shall consist of a single test class while the rest shall consist of more than one test class.

    • The suite thread pool is 2, so one suite will have to wait for one of the others to complete execution before it can begin execution
    • For one of the suites, the thread count and parallel mode are specified at the suite level
    • For one of the suites, the thread count and parallel mode are specified at the test level
    • For one of the suites, the parallel mode is specified at the suite level, and the thread counts are specified at the test level (thread counts for each test differ)
    • The thread count is less than the number of test methods for the tests in two of the suites, so some methods will have to wait for the active thread count to drop below the maximum thread count before they can begin execution.
    • The thread count is more than the number of test methods for the tests in one of the suites, ensuring that none of the methods in that suite should have to wait for any other method to complete execution

PTP-TC-3: Parallel by methods mode with sequential test suites using a non-parallel data provider but no dependencies and no factories.

VERIFIES:

That the methods are run for each data set given to them by the data provider in accordance with the thread count specified in parallel by methods mode. The data provider is NOT parallel, so each invocation of the method should in in the same thread.

SCENARIOS:

  1. Single suite with a single test consisting of a single test class with five methods with a data provider specifying 3 sets of data

    • Thread count and parallel mode are specified at the suite level
    • The thread count is equal to the number of test methods times 3, the number of times each method will be invoked with a data set from the data provider. Expectation is that only 5 threads will be spawned, one for each of the methods and that in this thread, the method will be invoked 3 times, once for each set of data from the data provider.

  2. Two suites with 1 and 2 tests respectively. One test for a suite shall consist of a single test class while the rest shall consist of more than one test class. Each test class has some methods with use a data provider and some which do not. Two data providers are used: one which provides two sets of data, one which provide three sets of data.

    • For both suites, the thread count and parallel mode are specified at the suite level
    • The thread count is less than the number of test methods for all tests in both suites, so methods will have to wait for the active thread count to drop below the maximum thread count before they can begin execution. The expectation is that a thread for a method does not terminate until the method has been invoked for all sets of data if it uses a data provider.

PTP-TC-4: Parallel by methods mode with parallel test suites using a non-parallel data provider but no dependencies and no factories.

VERIFIES:

That the methods are run for each data set given to them by the data provider in accordance with the thread count specified in parallel by methods mode. The data provider is NOT parallel, so each invocation of the method should in in the same thread.

SCENARIOS:

  1. Three suites with 1, 2 and 3 tests respectively. One test for a suite shall consist of a single test class while the rest shall consist of more than one test class. Some test classes have a mixture of some methods that use a data provider and some which do not. The data providers provide data sets of varying sizes.

    • The suite thread pool is 2, so one suite will have to wait for one of the others to complete execution before it can begin execution
    • For one of the suites, the thread count and parallel mode are specified at the suite level
    • For one of the suites, the thread count and parallel mode are specified at the test level
    • For one of the suites, the parallel mode is specified at the suite level, and the thread counts are specified at the test level (thread counts for each test differ)
    • The thread count is less than the number of test methods for the tests in two of the suites, so some methods will have to wait for the active thread count to drop below the maximum thread count before they can begin execution. The expectation is that a thread for a method does not terminate until the method has been invoked for all sets of data if it uses a data provider.
    • The thread count is more than the number of test methods for the tests in one of the suites, ensuring that none of the methods in that suite should have to wait for any other method to complete execution. The expectation is that a thread for a method does not terminate until the method has been invoked for all sets of data if it uses a data provider.

PTP-TC-5: Parallel by methods mode with sequential test suites using factories but no dependencies and no data providers.

VERIFIES:

That the methods are run for each test class instance created by a factory in accordance with the thread count specified in parallel by methods mode, and that methods in one instance do not wait for methods in another instance to complete before they execute if the thread count is less than the maximum.

SCENARIOS:

  1. Single suite with a single test consisting of a factory that provides two instances of a single test class with five methods

    • Thread count and parallel mode are specified at the suite level
    • The thread count is equal to the number of test methods times 2, the number of times each method will be invoked because there are two instances of the test class. Expectation is that 10 threads will be spawned because each method will run once for each of the two instances and that each instance of a test method will run in separate threads.

  2. Two suites with 1 and 2 tests respectively. One suite with two tests has a test consisting of a single test class without a factory while the other consists of factories which provide multiple instances of multiple test classes. One suite shall consist of a single test with multiple test classes which uses factories.

    • For both suites, the thread count and parallel mode are specified at the suite level
    • The thread count is less than the number of test methods for all tests in both suites, so methods will have to wait for the active thread count to drop below the maximum thread count before they can begin execution. The expectation is that threads will be spawned for each test method that executes just as they would if there were no factories and test suites simply used the default mechanism for instantiating single instances of the test classes.

PTP-TC-6: Parallel by methods mode with parallel test suites using factories but no dependencies and no data providers.

VERIFIES:

That the methods are run for each test class instance created by a factory in accordance with the thread count specified in parallel by methods mode, and that methods in one instance do not wait for methods in another instance to complete before they execute if the thread count is less than the maximum.

SCENARIOS:

  1. Three suites with 1, 2 and 3 tests respectively. One suite with two tests has a test consisting of a single test class without a factory while the other shall consist of factories which provide multiple instances of multiple test classes. One suite shall consist of a single test with multiple test classes which uses factories. One suite shall have multiple tests with multiple classes, none of which use a factory.

    • The suite thread pool is 2, so one suite will have to wait for one of the others to complete execution before it can begin execution
    • For one of the suites, the thread count and parallel mode are specified at the suite level
    • For one of the suites, the thread count and parallel mode are specified at the test level
    • For one of the suites, the parallel mode is specified at the suite level, and the thread counts are specified at the test level (thread counts for each test differ)
    • The thread count is less than the number of test methods for the tests in two of the suites, so some methods will have to wait for the active thread count to drop below the maximum thread count before they can begin execution.
    • The thread count is more than the number of test methods for the tests in one of the suites, ensuring that none of the methods in that suite should have to wait for any other method to complete execution
    • The expectation is that threads will be spawned for each test method that executes just as they would if there were no factories and test suites simply used the default mechanism for instantiating single instances of the test classes.

PTP-TC-7: Parallel by methods mode with sequential test suites using a factory with uses a non-parallel data provider and there are no dependencies.

VERIFIES:

  1. That the methods are run for each test class instance created by a factory for each data set given by the data provider in accordance with the thread count specified in parallel by methods mode
  2. That methods in one instance do not wait for methods in another instance to complete before they execute if the thread count is less than the maximum.

SCENARIOS:

  1. Single suite with a single test consisting of a single test class with five methods with a factory method using a data provider specifying 3 sets of data

    • Thread count and parallel mode are specified at the suite level
    • The thread count is equal to the number of test methods times 3, the number of times each method will be invoked with a data set from the data provider. Expectation is that only 15 threads will be spawned at once. There will be three instances of the test class, one for each of the three data sets and each instance will have the five test methods defined on the class, for a total of 15 test method invocations.
    • No test method invocation should be queued because the thread count is sufficient to service all the test method invocations.

  2. Two suites with 1 and 2 tests respectively. One suite with two tests has a test consisting of a single test class without a factory while the other consists of factories using data providers with varying numbers of data sets which provide multiple instances of multiple test classes. One suite shall consist of a single test with multiple test classes which use factories with data providers with varying numbers of data sets.

    • For both suites, the thread count and parallel mode are specified at the suite level
    • The thread count is less than the number of test methods for all tests in both suites, so methods will have to wait for the active thread count to drop below the maximum thread count before they can begin execution. The expectation is that threads will be spawned for each test method that executes just as they would if there were no factories and data providers and test suites simply use the default mechanism for instantiating single instances of the test classes.

PTP-TC-8: Parallel by methods mode with parallel test suites using factories with data providers but no dependencies.

VERIFIES:

  1. That the methods are run for each test class instance created by a factory for each data set provided by the factory's data provider in accordance with the thread count specified in parallel by methods mode, and that methods in one instance do not wait for methods in another instance to complete before they execute if the thread count is less than the maximum.
  2. That separate threads are spawned for each method and data set, not that the method is run for each data set in a single thread.

SCENARIOS:

  1. Three suites with 1, 2 and 3 tests respectively. One suite with two tests has a test consisting of a single test class without a factory while the other shall consist of test classes with factories using data providers with varying numbers of data sets. One suite shall consist of a single test with multiple test classes with factories using data providers with varying numbers of data sets. One suite shall have multiple tests with multiple classes, none of which use a factory.

    • The suite thread pool is 2, so one suite will have to wait for one of the others to complete execution before it can begin execution
    • For one of the suites, the thread count and parallel mode are specified at the suite level
    • For one of the suites, the thread count and parallel mode are specified at the test level
    • For one of the suites, the parallel mode is specified at the suite level, and the thread counts are specified at the test level (thread counts for each test differ)
    • The thread count is less than the number of test methods for the tests in two of the suites, so some methods will have to wait for the active thread count to drop below the maximum thread count before they can begin execution.
    • The thread count is more than the number of test methods for the tests in one of the suites, ensuring that none of the methods in that suite should have to wait for any other method to complete execution
    • The expectation is that threads will be spawned for each test method that executes just as they would if there were no factories and test suites simply used the default mechanism for instantiating single instances of the test classes.

PTP-TC-9: Parallel by methods mode with sequential test suites with no factories or data providers. There are method-on-method, intra-class group and inter-class group dependencies.

VERIFIES:

  1. That methods are executed only after the methods they depend on complete execution. This includes methods they directly depend on as well as methods belonging to groups they depend on.
  2. That the thread count is respected when the number methods that can execute is greater than the thread count
  3. That only methods without dependencies or methods with dependencies that have already completed execution are executed even if the thread count is larger than the the number of methods than can execute

SCENARIOS:

  1. Two suites with two tests each. One suite consists of only method-on-method dependencies and one of its tests contains a test class with no dependencies at all. The other suite has a test with both inter and intra-class group dependencies as well as a class with no dependencies at all. Its other test consists of classes with intra-class group dependencies only.

    • For one suite, the thread count is specified at the test level and for the other, the thread count is specified at the suite level.
    • One test has fewer methods available that can execute than its thread count due to dependencies when it starts, but the bottleneck clears and the maximum number of methods executes in a subsequent block of parallel methods
    • One test has more methods available that can execute than its thread count when it starts, but in a subsequent block of parallel methods, a bottleneck develops due to dependencies so fewer methods execute in parallel than the thread count would allow
Add a custom footer
Add a custom sidebar
Clone this wiki locally