Thanks for the report.

How do you run the class?

Then, there are 2 interpretations:

• no group means the default no-group, and the behavior is the expected one (before the no-group)
• no group is the default group, and the current behavior is an issue.

Related to #1574
Ping @krmahadevan

no group is no group.
I run this without any groups defined, with default IntelliJ IDEA runner both from context menu on test-method and from context menu on testng.xml:

<!DOCTYPE suite SYSTEM "http://testng.org/testng-1.0.dtd" >
<suite name="my-suite">
	<test name ="my-test">
		<packages>
			<package name="qa.ok.testng.*"/>
		</packages>
	</test>
</suite>

(package qa.ok.testng.* contains only one class "AlwaysRunTests" listed in my prev. comment)
No other settings/listeners/etc are present.
I've attached zip with pom, java and testng.xml:
always-run-test.zip

I reword:

no group is the default group

Your current suite will run all tests in the package, right?
And not just tests without group.

BTW, it is debatable and we can only quick fix regression for the moment.

Your current suite will run all tests in the package, right?

Yes. I understand. You mean my tests-list can also have some tests with groups so this can be the reason why Before-methods invoked.
But I don't want to run those methods if any of the 'parent' configurations failed.
In any case current behavior does not corresponds to documentation. I don't know if it is the documentation bug or it the bug in code which was introduced after documentation was published.
In documentation the parameter "alwaysRun" has 2 different meanings for before- and after- methods :

• For before methods (beforeSuite, beforeTest, beforeTestClass and beforeTestMethod, but not beforeGroups): If set to true, this configuration method will be run regardless of what groups it belongs to.
• For after methods (afterSuite, afterClass, ...): If set to true, this configuration method will be run even if one or more methods invoked previously failed or was skipped.

So first is about groups (always run regardless on group) and second - about failed/skipped (alwaysRun regardless on failed/skipped). But currently 'alwaysRun' for before-methods includes behavior for after-methods:
when set to true:

1. configuration method will be run even if one or more methods invoked previously failed or was skipped AND
2. configuration method will be run regardless of what groups it belongs to.

I've found this bug was introduced in testng-6.9.5 when fixing #420
Below the log for v6.9.4 (note that no other before methods were invoked except before-suite):

  D:\dev\AC\git\test-testng\testng.xml
I'm beforeSuite. I should be alone in console log. Going to fail...

java.lang.RuntimeException
	at qa.ok.testng.AlwaysRunTests.failedBeforeSuite(AlwaysRunTests.java:14)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.testng.internal.MethodInvocationHelper.invokeMethod(MethodInvocationHelper.java:85)
	at org.testng.internal.Invoker.invokeConfigurationMethod(Invoker.java:517)
	at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:213)
	at org.testng.internal.Invoker.invokeConfigurations(Invoker.java:140)
	at org.testng.SuiteRunner.privateRun(SuiteRunner.java:296)
	at org.testng.SuiteRunner.run(SuiteRunner.java:259)
	at org.testng.SuiteRunnerWorker.runSuite(SuiteRunnerWorker.java:52)
	at org.testng.SuiteRunnerWorker.run(SuiteRunnerWorker.java:86)
	at org.testng.TestNG.runSuitesSequentially(TestNG.java:1199)
	at org.testng.TestNG.runSuitesLocally(TestNG.java:1124)
	at org.testng.TestNG.run(TestNG.java:1032)
	at org.testng.IDEARemoteTestNG.run(IDEARemoteTestNG.java:72)
	at org.testng.RemoteTestNGStarter.main(RemoteTestNGStarter.java:123)
Test ignored.
===============================================
my-suite
Total tests run: 1, Failures: 0, Skips: 1
Configuration Failures: 1, Skips: 3
===============================================

Ok, thanks for digging 👍

@juherr - I am not sure how to go about fixing this because :

I am not able to figure out how to determine if the user ran with group filtering or if the user ran tests as is. The reason I say this is because a user may have applied a method selector (either as implementation or as a beanshell expression) to filter methods.

So if I cannot determine if the tests are being run by applying groups, I will not be able to determine if alwaysRun=true should be considered or not.

Guys, sorry but I just want to prevent breaking changes in testng, so will try to explain my opinion:
Before testng v6.9.5 parameter alwaysRun had very different meaning in @Before- and @After- methods. Below I tried to describe behavior for that version (based on official testng documentation, javadocs and implementation):

• In context of @BeforeSuite, @BeforeTest, @BeforeClass, @BeforeMethod :
  schedule this method for execution regardless to which group it belongs / does not belong (of cause if corresponding @Tests were scheduled)
  Please note: (by implementation up to 6.9.5):
  -- method will be skipped if any parent or preceding configurations failed.
  -- this behavior does not depend on config-failure-policy

• In context of @AfterSuite, @AfterTest, @AfterClass, @AfterMethod :
  (from javadoc and official documentation): configuration method will be run even if one or more methods invoked previously failed or was skipped

• In context of @Test:
  (from javadoc): run test-method even if it depends on a method that failed. This attribute ignored if test doesn't depend on any method or group.
  Please note:
  -- test-method will be skipped (will not run) if it depends on skipped methods.
  -- test-method will be skipped in every case when any of parent configurations failed.

Last note is very impotent. It means: even when if you force to execute @BeforeMethod after @BeforeClass was failed , all the methods which belongs to class will be skipped. So why we need to force execution of @BeforeMethod if test ignored in any way???

I think this behavior ( v6.9.5) should be saved because it was well-designed by @cbeust and it have meaning. There are no reasonable situations in testing when you should continue to execute 'child' prerequisites after 'parent' failed.

@krmahadevan, If I understand properly from PR 9d5e6d7, you want to force execution for before-methods only when configfailurepolicy = continue. Please use SRP principle for parameters in testng. Currently (v13.1) configfailurepolicy used to avoid multiple execution of the same configuration method. (Same means: method with concrete signature which belongs to concrete instance) Please don't use this parameter for any other purpose. This will break compatibility and will make process of testng configuration ugly.

Finally:
IMHO to fix current issue you should revert changes which were made in scope of #420 (328620f) - it was wrong fix. Maybe it fixed behavior in particular case but it does not fix root cause. I'm almost sure that issue #420 can be reproduced with or without alwaysRun on @BeforeSuite

@kool79 It looks you have a good knowledge of the issue. Maybe you can propose a pull request with the fix?

Sorry.. by mistake clicked the wrong button.

@kool79 Just want to "upvote" this issue: I agree with your assessment, and find the current behavior frustrating. Given your extensive knowledge of the issue, would you mind trying to submit a fix? (i.e., reverting #420 but also proposing a better fix for that?)

+1, our team affected by this issue.
Is there any work-around to force skip of before/after methods by manipulation with IContext for example?

any workaround for this issue?

@hg0605 - No there aren't any workarounds for this issue at the moment.

Any update on this? We want to terminate the suite at the BeforeSuite level, but are noticing all the beforeMethods are still running.