What new features would you like to have in TestNG?

[krmahadevan]

Have you ever wished that you had "feature x" in TestNG? Now is the time to call them out ?

Add some details of what you would like to have in TestNG in it's newer versions ?

[krmahadevan]

To get this started, TestNG should support Java Streams in it's data driven tests. This way we can now enhance our tests to have the @DataProvider annotated methods to start returning Stream<T>

[juherr]

@krmahadevan I think you should post on the mailing list if you want more answers

[krmahadevan]

@juherr Yes. Thats the plan. I will be sharing the link to this discussion on both LinkedIn and the TestNG users google forum

[ajmera25]

Tag support for the test.

Reason - In xml let say we have n number of test but we want to run subset of test matching to some condition. if we have tag mechanism then we already provided tag to the test and thus we can pass the tag from the command line without need to make any change to the xml or code. This is similar to what we can do in cucumber/karate.

[Travisnv]

After long time testng users, I find myself in a situation where testng fw just falls behind in term of making productivity with DI.

The intention with guice library is quite a "hard force" for me through various projects which turn me off.

I think we may want to make testng more friendly with DI other than guice.

[krmahadevan]

@Travisnv - If I u'stand the ask correctly, are you suggesting that TestNG should support the notion of Dependency Injection and allow the users to pick their DI framework of choice? If yes, then can you please help add more details in terms of what all you would expect. That will help us build a user story (or) multiple user stories around the dependency injection feature and start baking it into TestNG.

[krmahadevan]

@ajmera25 - I think you can perhaps take a look at the Beanshell capabilities that TestNG offers. This will help you achieve what you are asking as tag support in TestNG. Please check https://testng.org/#_beanshell_and_advanced_group_selection

[Siizy]

I am not sure if TestNg can generate a playlist for unit test while building a project just like NUnit, MsTest do. This way we have the ability to known what is going to run, run specific test from playlist on demand, save playlist and run it when needed. I understand it would require a plugin for each different IDE but this will be a great development unless we have the feature already.
Thanks

[krmahadevan]

@Siizy - Can you please help share some additional context (Screenshots or something on those lines) so that it's more clear as to what is the ask ? Also if this is something that we want from within an IDE, we can create an issue in the TestNG eclipse plugin repo. For the intellij plugin, I think we would need to create that on the JetBrains issue tracker page.

ping @missedone

[WasifIsrar]

Technically its not a feature request but a change request
If BeforeMethod is running for every test, the behaviour for every test should be different.
If it fails for first test but passes for next test, subsequent tests should not get skipped.

[krmahadevan]

@WasifIsrar - If this is a change request, then I would request you to please help file a bug with some code samples and expected output so that we can take a look at it.

[sureshguptaprf]

There should be a configurable option to treat the test run count as '1' or the data-size while using data provider.

When treating the TC count as '1', Final TC Status should be:
Fail - if atleast 1 fail
Skip - when all skip
Pass - when all pass

[krmahadevan]

@sureshguptaprf - Do you think that this ask should be confined to a specific data driven test, or if it should be something that should be controlled globally at the <test> (or) at the <suite> level ?

[sureshguptaprf]

Distributed execution capabilities i.e. splitting TCs across multiple nodes. Everyone who uses TestNG for their API Tests ends up running their tests on a single node with parallel threads. But it limits the performance when TCs are in thousands and need to be distributed across multiple nodes.

Alternatively, many create multiple suite files which are run on multiple machines in separate TestNG processes effectively making them separate runs and reported separately.

[krmahadevan]

Are you suggesting something like https://docs.gradle.com/enterprise/test-distribution/ ?

Distributed test execution would mean that there should be TestNG agents running on remote nodes and we would need to have some sort of a master/slave setup wherein tests submitted at the master node level (the master node could just be the surefire plugin JVM for e.g., which triggers test executions) would get distributed to the different nodes (which have been configured as well).

I know that in the past @cbeust has mentioned this.

ping @cbeust @juherr - Your thoughts on this ?

[juherr]

It was a project years ago but now TestNG is just a library that runs tests.
Managing nodes could be done by any external project like the one you pointed.

Tldr: not the responsability of testng

[missedone]

@sureshguptaprf i just googled that there's a commercial offering https://testgrid.io/blog/web-stories/testng-for-parallel-and-distributed-testing/

disclaim: i haven't tried it before

[sureshguptaprf]

Enriching it's Assertion APIs, making it fluent, which currently makes us look elsewhere for better Assertion libraries.

[krmahadevan]

@sureshguptaprf - With respect to assertions, I think it doesn't add a lot of value for TestNG to try and build parity with assertion specialised libraries such as AssertJ

Since assertJ is already a specialised assertions related library and provides for all the asks from assertions, it would make sense for TestNG users to just adopt it for exorbitant assertion asks.

@juherr - What do you think ?

[juherr]

Agree. The plan is more to remove the assertion api than improve it.

[prakashnarkhede]

I need an option to skip execution of @BeforeMethod against certain set of tests.
Example -
I have 10 tests, out of it, before 9 tests, i want to execute certain pre-requsite, which i will put under BeforeMethod, but say in 1 test, i dont need those pre-requsite to be executed, so want to skip execution o beforeMethod and AfterMethod for that specific test.
I know using DI, and some workaround this can be achieved, but i would like to see a very simple approach like @test(skipBeforeMethod=true) or something simillar

[prakashnarkhede]

Real-Time Analytics and Reporting Dashboard

Currently, most of us rely on extent reports, allure, or other tools in the market, You should also have agreed that testing reports is not up to the mark.
Looking for a feature that can get rid of the extra addition of a reporting library, making test automation a little more complex, and something that focuses on providing an advanced, real-time analytics and reporting dashboard within TestNG. It aims to offer detailed insights into test execution, performance metrics, and trend analysis, facilitating better decision-making and test management.

[krmahadevan]

@prakashnarkhede - This is currently already possible in TestNG.

Here are three options (Feel free to pick whichever works best for you)

For both these options, we are going to use a custom annotation that looks like below

import java.lang.annotation.Retention;
import java.lang.annotation.RetentionPolicy;
import java.lang.annotation.Target;

//Marker annotation
@Target(java.lang.annotation.ElementType.METHOD)
@Retention(RetentionPolicy.RUNTIME)
public @interface SkipConfiguration {}

Option 1 - Using Native Dependency Injection and Configuration method

Test class

import org.testng.SkipException;
import org.testng.annotations.BeforeMethod;
import org.testng.annotations.Test;

import java.lang.reflect.Method;

public class SampleTestClass {

    @BeforeMethod
    public void beforeMethod(Method method) {
        boolean shouldSkip = method.isAnnotationPresent(SkipConfiguration.class);
        String prefix = "Running configuration setup for ";
        if (shouldSkip) {
            prefix = "Skipping configuration setup for ";
        }
        System.err.println(prefix + method.getName());
        if (shouldSkip) {
            throw new SkipException(prefix);
        }
    }

    @Test
    public void testMethod1() {}

    @Test
    @SkipConfiguration
    public void testMethod2() {}
}

Option 2 - Using TestNG Listeners

Test class

import org.testng.annotations.Listeners;
import org.testng.annotations.Test;

@Listeners(SimpleListener.class)
public class AnotherSampleTestClass {

    @Test
    public void testMethod1() {}

    @Test
    @SkipConfiguration
    public void testMethod2() {}
}

Listener being used

import org.testng.*;

import java.lang.reflect.Method;
public class SimpleListener implements IInvokedMethodListener {

    @Override
    public void beforeInvocation(IInvokedMethod im, ITestResult tr) {
        ITestNGMethod tm = im.getTestMethod();
        Method method = tm.getConstructorOrMethod().getMethod();
        boolean shouldSkip = method.isAnnotationPresent(SkipConfiguration.class);
        String prefix = "Running configuration setup for ";
        if (shouldSkip) {
            prefix = "Skipping configuration setup for ";
        }
        System.err.println(prefix + method.getName());
        if (shouldSkip) {
            tr.setThrowable(new SkipException(prefix));
            tr.setStatus(ITestResult.SKIP);
        }
    }
}

Option 3 - Using IHookable

Base class

import org.testng.*;

import java.lang.reflect.Method;

public class AbstractTestClass implements IHookable {
    @Override
    public void run(IHookCallBack callBack, ITestResult tr) {
        ITestNGMethod tm = tr.getMethod();
        Method method = tm.getConstructorOrMethod().getMethod();
        boolean shouldSkip = method.isAnnotationPresent(SkipConfiguration.class);
        String prefix = "Running configuration setup for ";
        if (shouldSkip) {
            prefix = "Skipping configuration setup for ";
        }
        System.err.println(prefix + method.getName());
        if (shouldSkip) {
            throw new SkipException(prefix);
        }
        callBack.runTestMethod(tr);
    }
}

Test class

import org.testng.annotations.Test;

public class YetAnotherSampleTestClass extends AbstractTestClass {

    @Test
    public void testMethod1() {
    }

    @Test
    @SkipConfiguration
    public void testMethod2() {
    }
}

[krmahadevan]

@prakashnarkhede

Looking for a feature that can get rid of the extra addition of a reporting library, making test automation a little more complex, and something that focuses on providing an advanced, real-time analytics and reporting dashboard within TestNG. It aims to offer detailed insights into test execution, performance metrics, and trend analysis, facilitating better decision-making and test management.

Given that TestNG already provides you with handling TestNG results (either at the end via IReporter interface (or) on a more realtime basis via IInvokedMethodListener for e.g., I believe that you build this capability at your end without needing any additional changes into TestNG.

What you are asking for is more of a sophistication around test results analysis which I feel can be obtained if you use some sort of a tool such as https://reportportal.io/docs/

I am not sure if TestNG dev team (which is pretty lean at the moment) has the bandwidth to indulge into something like this.

@juherr - WDYT ?

[juherr]

I think reporting, except api, should be excluded from testng core.
It is an opportunity for 3rd parties and many projects already exist.

[ubutar]

What

I'd like to suggest at least some control over @DataProvider-annotated methods during retry. As discussed in #3041 dataprovider methods used to be invoked when retrying failed methods in v6, in v7 it was gradually removed until 7.8.0 when it was removed finally, so that tests are always fed the original data generated one time.

Why

• Dataproviders are frequently used as generators, which are meant to provide unique data, due to constraints of application under test. Retrying with already invalid data makes little sense
• Dataproviders with single object returned are a frequent case out there.
• It used to work like that in v6 after all
• Painful to migrate from 6 to 7

How

What I'd like to get is something similar to:

@DataProvider(alwayRun="true")
public Iterator<Object[]> method() {...}

or

@DataProvider
@Generator
public Iterator<Object[]> method() {...}

or

@Generator
public Iterator<Object[]> method() {...}

or

@DataProvider(generator="true")
public Iterator<Object[]> method() {...}

or anything which would rerun annotated method everytime a test is retried, e.g. using IRetryAnalyzer or any other retry mechanism. I guess alwaysRun=true is the most obvious choice.

For the case of NxM data object, if N1<N tests failed and are to be retried, I still suggest generating whole set of data by running the method, but pick only those data line where tests failed. For parallel execution on dataProvider level it may get tricky getting the order, but I guess internal indexing of returned objects might help (Sorry if it's already the case, I didn't check)

Things to consider

@Factory

[baflQA]

I would be happy to see more ways of controlling the Before/AfterMethod configuration.
I would e.g. want to have an easy way of skipping the first execution of the BeforeMethod, and the last execution of the AfterMethod.
E.g. - i want to refresh the page before/after the test, and it doesn't make sense to do that right after the test setup and the initial navigation, or before the cleanup and the browser shut down.

[krmahadevan]

@baflQA can you please help share a sample which contains the beforeMethod and afterMethod along with 2 or more test methods ( please include regular tests, data driven tests and tests with invocation count ) and share the expected output.

That way the example would make it very clear in terms of the expectations

[baflQA]

Hey @krmahadevan - i was thinking about something like this:
https://github.com/baflQA/testNG_parallel_debug/blob/master/src/test/java/testngparallel/SkipFirstBeforeMethodExecution.java

[krmahadevan]

@baflQA - You should be able to do this even today. Here's a sample that shows this

import org.testng.annotations.BeforeMethod;
import org.testng.annotations.Test;

import java.lang.reflect.Method;
import java.util.concurrent.atomic.AtomicBoolean;

public class SampleTestCase {

    private final AtomicBoolean once = new AtomicBoolean(false);

    @BeforeMethod
    public void beforeMethod(Method method) {

        if (once.compareAndSet(false, true)) {
            //do nothing but return
            System.err.println("Skipping config method the first time for " + method.getName());
        } else {
            System.err.println("Running setup for " + method.getName());
            //simulate the setup
        }
    }

    @Test
    public void testMethod1() {}

    @Test
    public void testMethod2() {}

    @Test
    public void testMethod3() {}
}

[krmahadevan]

As a user I should be able to associate context to listener invocation.

Currently TestNG allows me to define listeners globally in a direct way (via service loaders or via the <listener> tag) (or) in an indirect way (via the annotation @Listeners). But TestNG always hoists these listeners at the global level and then applies them across the suites it finds for execution.

Maybe it would make sense for us to apply the listeners defined via the @LIsteners ONLY to the class on which it was annotated and maybe the child classes.

For e.g, lets say there's a test class that looks like below

@Listeners({DataProviderListener.class})
public class MyTestCase {
// more test code
}

TestNG should invoke the listener DataProviderListener only to MyTestCase and its sub-classes.

[X-NoNAME]

1. It would be great if we could change the status of the @ Before... methods invoke result (in the afterInvocation method)

2. Add the ability to set a timeout for config methods even if IConfigurable interface used

[bobshie]

There is an requirement we discussed before.

A DataProvider is defined that include a data set (A, B ,C ,D, E ... ) and we want to launch test cases testFunc(id) in multithreads (max thread is 3).
what we expect is:
testFunc(A) testFunc(B) testFunc(C) in parallel.
and then, after some test finished, it will create a new thread to run D, and then E, F....

interface is like:

@DataPoolProvider(max_thread = 3, dataprovider=dataset)
public void testFunc(id){
}

[krmahadevan]

The dataprovider-thread-count attribute at the suite level already lets you control the pool size. Does that not work for you? Or are you trying to control the thread pool size on a per data driven test method ?

[bobshie]

There is request to control the thread pool size based on the data. for there are some resource limit based on some data.

[krmahadevan]

@bobshie - There are some experimental features that lets you define a common thread pool for running all your data providers or running both your regular and data driven tests in a shared thread pool

See here : https://testng.org/#_controlling_threadpool_usage

PS: If you use a global thread pool in which all your regular and data driven tests will run, you may experience this issue

[bobshie]

Do we have a page that include all features that provides via TestNG specific version? I want to check what big features that TestNG 5.0 to 7.0 can provide and then I maybe use these features in my project to save some code or make test cases simple.

[krmahadevan]

@bobshie - unfortunately there's no such page that lists out features by version.