Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Programmatic Loading of Tests in External Jar #1999

Closed
1 of 8 tasks
ghost opened this issue Jan 6, 2019 · 23 comments
Closed
1 of 8 tasks

Programmatic Loading of Tests in External Jar #1999

ghost opened this issue Jan 6, 2019 · 23 comments

Comments

@ghost
Copy link

ghost commented Jan 6, 2019

TestNG Version

6.14.3

Expected behavior

TestNG runs tests using added ClassLoader.

Actual behavior

TestNG not finding tests in ClassLoader.

Is the issue reproductible on runner?

  • Shell
  • Maven
  • Gradle
  • Ant
  • Eclipse
  • IntelliJ
  • NetBeans
  • Java code

Test case sample

XmlSuite suite = new XmlSuite();
suite.setName("Invoked Run");

XmlTest test = new XmlTest();
test.setName("Invoked Run");
test.setParallel(XmlSuite.ParallelMode.NONE);
test.setXmlPackages(Collections.singletonList(new XmlPackage("com.tests.*")));
test.setTimeout(TimeUnit.SECONDS.toMillis(300));

TestNG testNg = new TestNG();
testNg.setOutputDirectory(persisPath);
testNg.setXmlSuites(Collections.singletonList(suite));

URLClassLoader customClassLoader = URLClassLoader.newInstance(jarUrls);
testNg.addClassLoader(customClassLoader);

testNg.run();
int status = testNg.getStatus();

When I run debug and hit the customClassLoader, I can run this successfully:

customClassLoader.loadClass("com.tests.SampleTest");

It retrieves the annotations appropriately as well.

@krmahadevan
Copy link
Member

@revof11 - Is this issue related to this stack overflow post ?

Also can you please try again using TestNG 7.0.0-beta2 and post back if you still see the problem ?
http://repo1.maven.org/maven2/org/testng/testng/7.0.0-beta2/

If the problem still exists, request you to please provide a complete sample that can be used to reproduce the issue. The code you shared does not show how the jars were created and the contents of the jar as well.

@ghost
Copy link
Author

ghost commented Jan 6, 2019

I'll refactor it to see what I can do with beta2. Looks like I'll need to refactor a bit. When I pulled the dependency from Maven central it yelled at me about the org.testng.TestNG class not being available but it doesn't complain with beta1. I'll do beta1 and then refactor into beta2.

Looks like in beta 1 I just need to fix:

[TestNG] Reporter org.testng.reporters.jq.Main@67ec72aa failed
java.lang.NullPointerException
at org.testng.reporters.jq.TimesPanel.maxTime(TimesPanel.java:124)
at org.testng.reporters.jq.TimesPanel.getContent(TimesPanel.java:89)
at org.testng.reporters.jq.BaseMultiSuitePanel.generate(BaseMultiSuitePanel.java:25)
at org.testng.reporters.jq.Main.generateReport(Main.java:74)
at org.testng.TestNG.generateReports(TestNG.java:1027)
at org.testng.TestNG.run(TestNG.java:970)

@krmahadevan
Copy link
Member

@revof11 - 7.0.0-beta2 was a bad build. Please make use of 7.0.0-beta3 and retry.

@krmahadevan
Copy link
Member

ping @revof11 - Any updates on this ?

@krmahadevan
Copy link
Member

I have retried this using TestNG 7.0.0-beta5. I am not able to reproduce the issue.

I built a standalone jar which contains just 1 test class [ its a mavenised project wherein the test class resided under src/main/java ]

Here's how the test class looks like (this is what resides in the jar)

package com.rationaleemotions;

import org.testng.Reporter;
import org.testng.annotations.Test;

public class AppTest {

  @Test
  public void testMethod() {
    Reporter.log("Hello world", true);
  }

}

Here's the method

  private static void runTests(File jar) throws MalformedURLException {
    TestNG testNg = new TestNG();
    URLClassLoader customClassLoader = URLClassLoader.newInstance(new URL[]{jar.toURI().toURL()});
    testNg.addClassLoader(customClassLoader);
    XmlSuite suite = new XmlSuite();
    suite.setName("Invoked Run");

    XmlTest test = new XmlTest();
    test.setXmlSuite(suite);
    test.setName("Invoked Run");
    test.setParallel(XmlSuite.ParallelMode.NONE);
    XmlClass xmlClass = new XmlClass("com.rationaleemotions.AppTest");
    test.setClasses(Collections.singletonList(xmlClass));
    suite.setTests(Collections.singletonList(test));
    System.err.println(suite.toXml());

    testNg.setXmlSuites(Collections.singletonList(suite));

    testNg.setVerbose(2);
    testNg.run();
    int status = testNg.getStatus();
    System.err.println("Status ===> " + status);
  }

Output

Jar Location : /Users/krmahadevan/temp/issue_1999/target/issue_1999-1.0-SNAPSHOT.jar
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd">
<suite name="Invoked Run">
  <test thread-count="5" name="Invoked Run">
    <classes>
      <class name="com.rationaleemotions.AppTest"/>
    </classes>
  </test> <!-- Invoked Run -->
</suite> <!-- Invoked Run -->

Hello world
PASSED: testMethod

===============================================
    Invoked Run
    Tests run: 1, Failures: 0, Skips: 0
===============================================


===============================================
Invoked Run
Total tests run: 1, Passes: 1, Failures: 0, Skips: 0
===============================================

Status ===> 0

Process finished with exit code 0

@akshaybadgujarjs
Copy link

@krmahadevan Have you tried this by executing one jar into another using command line arguments ?
For me it is failing.!

@krmahadevan
Copy link
Member

@akshaybadgujarjs - What version are you using ?

@akshaybadgujarjs
Copy link

I'm using 7.0.0

My use case is I wanted to create one executor service which can take up any jar and be able to execute any test present in 2nd jar. I have developed simple application and trying to execute the same. Please find github repo for demo and test repos
Demo - is executor service which triggers execution
test - contains sample test

https://github.com/akshaybadgujarjs/demo
https://github.com/akshaybadgujarjs/test

Please let me know in case any more information required.

@krmahadevan
Copy link
Member

The latest released version of TestNG is 7.6.1 (Needs JDK11). Can you please quickly give it a try using this and let me know if its still a problem? Based on your test result, I will try to dig in further and find out what is going on.

@akshaybadgujarjs
Copy link

Sure will try and get back to you.!

@akshaybadgujarjs
Copy link

Faced same issue with updated test ng version as well. Log text file attached for your reference.

logs.txt

@krmahadevan
Copy link
Member

@akshaybadgujarjs - Sounds like you are facing #2770
This has been fixed in 7.6.2-SNAPSHOT and should be available in the upcoming 7.6.2 released version.

@akshaybadgujarjs
Copy link

My concern is not with java.nio.file.FileAlreadyExistsException: test-output/testng-reports.css this exception.
My query is that my test method is not getting executed at all.!
Pulsar Tests Suite
Total tests run: 0, Passes: 0, Failures: 0, Skips: 0

@krmahadevan
Copy link
Member

ah ok! Yeah. My bad. I got sidelined with that exception. I will take a look at this and let you know what is going on.

@akshaybadgujarjs
Copy link

No problem. Thanks for your assistance.!

@krmahadevan
Copy link
Member

@akshaybadgujarjs - I can't reproduce the problem. The code works fine for me.

I cloned both your repositories and then built your test project.

After that I applied this patch

patch
diff --git a/build.gradle b/build.gradle
index 2905641..221d411 100644
--- a/build.gradle
+++ b/build.gradle
@@ -2,6 +2,7 @@ plugins {
 	id 'java'
 	id 'org.springframework.boot' version '2.4.13'
 	id 'io.spring.dependency-management' version '1.0.15.RELEASE'
+	id 'idea'
 }
 
 group = 'com.example'
@@ -21,8 +22,7 @@ dependencies {
 	implementation 'org.springframework.boot:spring-boot-starter-web'
 	testImplementation 'org.springframework.boot:spring-boot-starter-test'
 	// https://mvnrepository.com/artifact/org.testng/testng
-//     implementation group: 'org.testng', name: 'testng', version: '7.5'
-    implementation group: 'org.testng', name: 'testng', version: '7.0.0'
+    implementation group: 'org.testng', name: 'testng', version: '7.6.1'
 
     compileOnly "org.projectlombok:lombok:${lombokVersion}"
     annotationProcessor "org.projectlombok:lombok:${lombokVersion}"
@@ -36,5 +36,5 @@ tasks.named('test') {
 }
 
 bootJar {
-    mainClass = 'com.example.demo.DemoApplication'
-}
\ No newline at end of file
+    mainClass.set('com.example.demo.DemoApplication')
+}
diff --git a/src/main/java/com/example/demo/SampleController.java b/src/main/java/com/example/demo/SampleController.java
index b3d753b..7cd2788 100644
--- a/src/main/java/com/example/demo/SampleController.java
+++ b/src/main/java/com/example/demo/SampleController.java
@@ -1,5 +1,6 @@
 package com.example.demo;
 
+import java.net.MalformedURLException;
 import org.springframework.beans.factory.annotation.Autowired;
 import org.springframework.web.bind.annotation.GetMapping;
 import org.springframework.web.bind.annotation.RestController;
@@ -7,11 +8,11 @@ import org.springframework.web.bind.annotation.RestController;
 @RestController
 public class SampleController {
 
-    @Autowired
-    private TestRunner testRunner;
+  @Autowired
+  private SimpleTestRunner simpleTestRunner;
 
-    @GetMapping("/execute-tests")
-    void executeTests(){
-        testRunner.triggerTestFromRequest();
-    }
+  @GetMapping("/execute-tests")
+  public void executeTests() throws MalformedURLException {
+    simpleTestRunner.runTestCase();
+  }
 }
diff --git a/src/main/java/com/example/demo/SimpleTestRunner.java b/src/main/java/com/example/demo/SimpleTestRunner.java
new file mode 100644
index 0000000..6938de0
--- /dev/null
+++ b/src/main/java/com/example/demo/SimpleTestRunner.java
@@ -0,0 +1,74 @@
+package com.example.demo;
+
+import java.io.File;
+import java.net.MalformedURLException;
+import java.net.URL;
+import java.net.URLClassLoader;
+import java.util.Collections;
+import lombok.extern.slf4j.Slf4j;
+import org.springframework.beans.factory.annotation.Configurable;
+import org.springframework.stereotype.Component;
+import org.springframework.stereotype.Service;
+import org.testng.IInvokedMethod;
+import org.testng.IInvokedMethodListener;
+import org.testng.ITestResult;
+import org.testng.TestNG;
+import org.testng.xml.XmlClass;
+import org.testng.xml.XmlSuite;
+import org.testng.xml.XmlTest;
+
+@Slf4j
+@Service
+@Component
+@Configurable
+public class SimpleTestRunner {
+
+  private static final String JAR =
+      "/Users/krishnamahadevan/temp/testng-1999/test/build/libs/test-0.0.1-SNAPSHOT.jar";
+
+  public void runTestCase() throws MalformedURLException {
+    TestNG testNg = new TestNG();
+
+    URLClassLoader customClassLoader = URLClassLoader.newInstance(new URL[]{getURL()});
+    testNg.addClassLoader(customClassLoader);
+    XmlSuite suite = new XmlSuite();
+    suite.setName("Pulsar Tests Suite");
+
+    XmlTest test = new XmlTest();
+    test.setXmlSuite(suite);
+    test.setName("Pulsar Test");
+    test.setParallel(XmlSuite.ParallelMode.NONE);
+    XmlClass xmlClass = new XmlClass("com.example.test.PulsarTestsSample");
+    test.setClasses(Collections.singletonList(xmlClass));
+    test.setVerbose(2);
+    suite.setTests(Collections.singletonList(test));
+    log.info(suite.toXml());
+    testNg.setXmlSuites(Collections.singletonList(suite));
+
+    testNg.addListener(new LocalListener());
+    testNg.setVerbose(2);
+    testNg.run();
+    int status = testNg.getStatus();
+    log.info("Status ===> {} ", status);
+  }
+
+
+  private static URL getURL() {
+    try {
+      URL url = new File(JAR).toURI().toURL();
+      log.info("Working with URL {}", url);
+      return url;
+    } catch (MalformedURLException e) {
+      log.error("Could not load URL. Root cause: {}", e.getMessage(), e);
+      throw new RuntimeException(e);
+    }
+  }
+
+  public static class LocalListener implements IInvokedMethodListener {
+
+    @Override
+    public void beforeInvocation(IInvokedMethod method, ITestResult testResult) {
+      log.info("About to run {}", method.getTestMethod().getQualifiedName());
+    }
+  }
+}

Now when I ran your main application and then triggered a get on http://localhost:8080/execute-tests, I noticed that the test runs perfectly fine.

I have attached the logs.

logs.txt

@akshaybadgujarjs
Copy link

I will give it a try and confirm.

@akshaybadgujarjs
Copy link

akshaybadgujarjs commented Dec 1, 2022

Hello @krmahadevan

I checked it. It is working fine in IDE. I suppose you have executed it from IDE as well. Is that correct ? Because from IDE even I didn't faced any issue previously.
The main problem occurs when I try to execute it from command line. If you can reproduce please let me know.!

FYI, I'm using given command to create build and run it
==> To Build
./gradlew clean build -x test

==> To Run
java -agentlib:jdwp=transport=dt_socket,server=y,suspend=n,address=5010 -jar -Dspring.profiles.active=local /Users/akshay.badgujar_js/IdeaProjects/demo/build/libs/demo-0.0.1-SNAPSHOT.jar

commnad line logs.txt

@krmahadevan
Copy link
Member

I checked it. It is working fine in IDE. I suppose you have executed it from IDE as well. Is that correct ? Because from IDE even I didn't faced any issue.

I didnt see you mention anywhere on the mode of running it. So I ran it from within the IDE.

I was able to get the CLI running by using

java -cp build/libs/demo-0.0.1-SNAPSHOT.jar -Dloader.path=file:/Users/krishnamahadevan/temp/testng-1999/test/build/libs/ org.springframework.boot.loader.PropertiesLauncher

Refer here

@akshaybadgujarjs
Copy link

@krmahadevan Have you tried this by executing one jar into another using command line arguments ? For me it is failing.!

Actually it was my first question about running it via command line.
Is your test executed after running it via CLI ?
For me it is still status 8 even after using your method for jar execution.
Running jar is not an issue. Once I calls for API, tests does not get executed. It gives me status 8 instead of 0 thats the concern.
Logs attached for your reference.

CLI Run with status 8.txt

@krmahadevan
Copy link
Member

@akshaybadgujarjs - It works for me

This is the command that I am using:

java -cp build/libs/demo-0.0.1-SNAPSHOT.jar -Dloader.path=file:/Users/krishnamahadevan/temp/testng-1999/test/build/libs/ org.springframework.boot.loader.PropertiesLauncher
2022-12-01 11:55:31.946  INFO 18916 --- [nio-8080-exec-1] com.example.demo.SimpleTestRunner        : Working with URL file:/Users/krishnamahadevan/temp/testng-1999/test/build/libs/test-0.0.1-SNAPSHOT.jar
2022-12-01 11:55:31.966  INFO 18916 --- [nio-8080-exec-1] com.example.demo.SimpleTestRunner        : <?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE suite SYSTEM "https://testng.org/testng-1.0.dtd">
<suite name="Pulsar Tests Suite">
  <test thread-count="5" name="Pulsar Test" verbose="2">
    <classes>
      <class name="com.example.test.PulsarTestsSample"/>
    </classes>
  </test> <!-- Pulsar Test -->
</suite> <!-- Pulsar Tests Suite -->

2022-12-01 11:55:32.025  INFO 18916 --- [nio-8080-exec-1] com.example.test.PulsarTestsSample       : PulsarTestsSample constructor
2022-12-01 11:55:32.079  INFO 18916 --- [nio-8080-exec-1] com.example.demo.SimpleTestRunner        : About to run com.example.test.PulsarTestsSample.sampleTest
2022-12-01 11:55:32.082  INFO 18916 --- [nio-8080-exec-1] com.example.test.PulsarTestsSample       : sampleTest
PASSED: sampleTest
        Add Restaurant Slot With No Holiday Slot And Verify

===============================================
    Pulsar Test
    Tests run: 1, Failures: 0, Skips: 0
===============================================


===============================================
Pulsar Tests Suite
Total tests run: 1, Passes: 1, Failures: 0, Skips: 0
===============================================

2022-12-01 11:55:32.181  INFO 18916 --- [nio-8080-exec-1] com.example.demo.SimpleTestRunner        : Status ===> 0 

@akshaybadgujarjs
Copy link

Okay thank you @krmahadevan for all the assistance and help here🙂
Will check and confirm.!

@akshaybadgujarjs
Copy link

It worked as expected. Thank you so much.! 🙂

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants