Add Kotlin contracts to exposed Kotlin API #3259

awelless · 2023-04-17T13:25:11Z

Overview

Resolves #1866.
Introduced assertNull, assertNotNull and assertInstanceOf methods.
Introduced contracts for assertNull, assertNotNull, assertThrows and assertDoesNotThrow methods

I hereby agree to the terms of the JUnit Contributor License Agreement.

Definition of Done

There are no TODOs left in the code
Method preconditions are checked and documented in the method's Javadoc
Coding conventions (e.g. for logging) have been followed
Change is covered by automated tests including corner cases, errors, and exception handling
Public API has Javadoc and @API annotations
Change is documented in the User Guide and Release Notes

marcphilipp · 2023-04-22T12:43:17Z

junit-jupiter-api/junit-jupiter-api.gradle.kts

+kotlin {
+	sourceSets {
+		main {
+			languageSettings.optIn("kotlin.contracts.ExperimentalContracts")


Do consumers of junit-jupiter-api also have to opt-in? If so, what happens of they don't?

No, since we opt-in in the junit-jupiter-api, consumers don't need to specify any opt-ins in their code

junit-jupiter-api/src/main/kotlin/org/junit/jupiter/api/Assertions.kt

JLLeitschuh · 2023-05-26T14:08:41Z

junit-jupiter-api/src/main/kotlin/org/junit/jupiter/api/Assertions.kt

I think fun fail(message: (() -> String)?) should also have a contract added to it?

Also, can contracts be added to vararg executables: () -> Unit?

callsInPlace supports only non-nullable arguments.
It would be great to have something like:

fun fail(message: (() -> String)?): Nothing { contract { if (message != null) { callsInPlace(message, EXACTLY_ONCE) } } Assertions.fail<Nothing>(message) }

but if is not allowed in a contract definition.

Similar thing applies to vararg executables: () -> Unit

Could the if be partially resolved like this? So that more (most) calls end up using the contract description for analysis?

// Covers method references `fail(::foo)`, inline lambdas `fail { }`. fun fail(message: () -> String): Nothing { contract { callsInPlace(message, EXACTLY_ONCE) } Assertions.fail<Nothing>(message) } // Covers backwards compatibility and potential edge cases like `fail(foo.bar)` where bar is a nullable functional type. fun fail(message: (() -> String)?): Nothing { Assertions.fail<Nothing>(message) }

The method signatures are same from JVM perspective.

Platform declaration clash: The following declarations have the same JVM signature (fail(Lkotlin/jvm/functions/Function0;)Ljava/lang/Void;): fun fail(message: (() -> String)?): Nothing defined in org.junit.jupiter.api in file Assertions.kt fun fail(message: () -> String): Nothing defined in org.junit.jupiter.api in file Assertions.kt

Huh, I didn't get that when I tried. Anyway, @JvmName usually helps. The question is if this is a feasible solution to get contract in for most, and still keep supporting other edge cases. What's the use case for nullable lambda here?

Yes, creating a separate method with a contract and non-nullable lambda allows kotlin compiler to carry out contract checks when it's sure the passed lambda is not null.
I assume, lambda is made nullable to resemble existing java api where messageSupplier can be nullable

callsInPlace for fail method is also specified with InvocationKind.UNKNOWN .
If we set it to EXACTLY_ONCE the following code compiles:

val expectedValue = "value" val value: String try { fail { value = expectedValue "message" } } catch (_: AssertionFailedError) { value = "another value" } assertEquals(expectedValue, value)

Here we reassign value albeit it is val. I reckon it isn't desirable to allow this code to compile

TWiStErRob · 2023-05-26T14:12:36Z

junit-jupiter-api/src/main/kotlin/org/junit/jupiter/api/Assertions.kt

-    Assertions.assertTimeoutPreemptively(timeout, executable)
+fun <R> assertTimeoutPreemptively(timeout: Duration, executable: () -> R): R {
+    contract {
+        callsInPlace(executable, EXACTLY_ONCE)


Is it actually true for things that can/expected to throw?
(I have a feeling we can't say "exactly once", but I might be misinterpreting the enum.)
Examples where it would not have compiled without the contract, and now it's a runtime error:

val value: String // Swapped version of fun `assertThrows with value initialization in lambda`() in this PR assertThrows<AssertionError> { Assertions.fail("message") value = "string" } assertEquals("string", value) // AssertionFailedError: expected: <string> but was: <null>

val x: String assertThrows<IOException> { File("//").readText() // Deterministic IOException due to invalid name. x = "abc" // Will never execute, because it always throws. } // Kotlin thinks x is initialized here, because of the EXACTLY_ONCE. assertEquals(3, x.length) // Consistent NPE, due to "effectively unreachable code" above.

val x: String assertThrows<AssertionError> { assertTimeoutPreemptively(Duration.seconds(1)) { Thread.sleep(2000) x = "" // Will never execute, because it always times out. } } // Kotlin thinks x is initialized here, because of the chain of EXACTLY_ONCE. println(x.length) // Consistent NPE, due to "effectively unreachable code" above.

What happens if you write this in Kotlin with try/catch? I presume the compiler pitches a fit?

val x: String try { File("//").readText() // Deterministic IOException due to invalid name. x = "abc" // Will never execute, because it always throws. } catch (e: IOException) { } assertEquals(3, x.length)

What about this?

val x: String File("//").apply { readText() // Deterministic IOException due to invalid name. x = "abc" // Will never execute, because it always throws. } assertEquals(3, x.length)

If the apply case fails in the same way as your example, I think it's probably fine?

This looks like it will compile fine:

import java.io.File var x: String File("//").apply { readText() // Deterministic IOException due to invalid name. x = "abc" // Will never execute, because it always throws. } if (3 == x.length) { }

Huh, very good point! 😁

try-catch example will not compile, but because not all code paths initialize x, but the point is valid there too, you can resolve this thread (I can't).

You could theoretically reimplement this logic, in kotlin, and expose it as an inline function. Then the try/catch will be inlined. Then there will be no need for the contract

junit5/junit-jupiter-api/src/main/java/org/junit/jupiter/api/AssertThrows.java

Lines 49 to 74 in ffab3a3

private static <T extends Throwable> T assertThrows(Class<T> expectedType, Executable executable,

Object messageOrSupplier) {

try {

executable.execute();

}

catch (Throwable actualException) {

if (expectedType.isInstance(actualException)) {

return (T) actualException;

}

else {

UnrecoverableExceptions.rethrowIfUnrecoverable(actualException);

throw assertionFailure() //

.message(messageOrSupplier) //

.expected(expectedType) //

.actual(actualException.getClass()) //

.reason("Unexpected exception type thrown") //

.cause(actualException) //

.build();

}

}

throw assertionFailure() //

.message(messageOrSupplier) //

.reason(format("Expected %s to be thrown, but nothing was thrown.", getCanonicalName(expectedType))) //

.build();

}

Contracts don't work well when we suppress exception. Issues one and two.
To prevent any kind of errors and confusion, I suggest to replace callsInPlace(executable, EXACTLY_ONCE) with callsInPlace(executable) in all assertThrows methods.

What do you think?

Makes sense, I would be explicit with second param though, just for readability. If the Kotlin compiler is improved, is it easy to swap to a "better exactly once"? Or would it break users?

I haven't noticed any improvements in contracts. InvocationKind.UNKNOWN is set for assertTimeoutPreemptively methods because of the same reasons as for assertThrows methods

TWiStErRob · 2023-05-26T14:30:40Z

junit-jupiter-engine/src/test/kotlin/org/junit/jupiter/api/KotlinAssertionsTests.kt

+
+        assertThrows<AssertionError> {
+            value = "string"
+            Assertions.fail("message")


Should this just be

Suggested change

Assertions.fail("message")

fail("message")

considering this is Kotlin code and there's a : Nothing signature top level fail.

Contracts work really weird with Nothing return type. Compiler warns that expression after assertThrows is not reachable while it is.

In the thread above I proposed not to specify InvocationKind for assertThrows methods. If there's no InvocationKind specified, value assignment in lambda will be forbidden and this test will make no sense

In that case I agree with UNKNOWN for invocation kind, because not being able to call production code that declares Nothing as return type is a big restriction. The point of assertThrows is that it tests exceptions thrown from a method, and Nothing declares exactly that.

In such case I'll set InvocationKind.UNKNOWN for assertThrows methods

junit-jupiter-api/src/main/kotlin/org/junit/jupiter/api/Assertions.kt

Added assertNull and assertNotNull methods with contracts. Added contracts for assertThrows and assertDoesNotThrow methods. assertInstanceOf can be implemented only with kotlin 1.4, because refined generics [are not supported](https://youtrack.jetbrains.com/issue/KT-28298) in contracts for kotlin 1.3 yet. Issue: junit-team#1866

Kotlin assertNull and assertNotNull methods are marked as experimental. Issue: junit-team#1866

Lambda invoked in assertThrows is marked as being called UNKNOWN number of times, since contracts do nothing with exception suppression Issue: junit-team#1866

Created another fail method with a non-nullable lambda parameter Issue: junit-team#1866

TWiStErRob

Looks to be in a reasonable state. Have some thoughts though.

junit-jupiter-api/src/main/kotlin/org/junit/jupiter/api/Assertions.kt

TWiStErRob · 2023-08-10T16:53:28Z

junit-jupiter-api/src/main/kotlin/org/junit/jupiter/api/Assertions.kt

-    Assertions.assertTimeoutPreemptively(timeout, executable)
+fun <R> assertTimeoutPreemptively(timeout: Duration, executable: () -> R): R {
+    contract {
+        callsInPlace(executable, EXACTLY_ONCE)


Makes sense, I would be explicit with second param though, just for readability. If the Kotlin compiler is improved, is it easy to swap to a "better exactly once"? Or would it break users?

Because of kotlin smart casts, there is no need to return a non-nullable value from `assertNotNull` methods Issue: junit-team#1866

Api version is increased to 5.11 for `assertNull` and `assertNotNull` methods Issue: junit-team#1866

Added `assertInstanceOf` assertions with upcasting contracts Issue: junit-team#1866

junit-jupiter-api/src/main/kotlin/org/junit/jupiter/api/Assertions.kt

Issue: junit-team#1866

TWiStErRob

Did a full review, wrote down what I noticed. Feel free to ignore some of them :)

Please resolve the conversations that are resolved, I can't do it, because only the PR Author and repo Members can.

And then we'll probably need a round of review from Marc or Jonathan.

TWiStErRob · 2023-08-27T15:23:32Z

junit-jupiter-api/src/main/kotlin/org/junit/jupiter/api/Assertions.kt

+ * @see Assertions.fail
+ */
+@JvmName("fail_nonNullableLambda")
+fun fail(message: () -> String): Nothing {


JUnit Experimental?

TWiStErRob · 2023-08-27T15:27:41Z

junit-jupiter-api/src/main/kotlin/org/junit/jupiter/api/Assertions.kt

+inline fun <reified T : Throwable> assertThrows(message: String, executable: () -> Unit): T {
+    contract {
+        callsInPlace(executable, UNKNOWN)
+    }


Is it possible to test for this contract?
As in: if it's removed, something should stop compiling or fail a test, right?
Probably the one in the related convo: #3259 (comment)

(This is probably true for all of them.)

I haven't found a way to test callsInPlace(..., UNKNOWN) contracts.
I though having another function with a contract inside could work, but it seems, kotlin compiler doesn't check whether a function is really called in place in the functions it's passed to.

fun notWorkingTest(value: () -> Unit) { contract { callsInPlace(value, UNKNOWN) } assertThrows<AssertionError>(value) // works fine even when assertThrows doesn't have a contract }

TWiStErRob · 2023-08-27T15:29:26Z

junit-jupiter-engine/src/test/kotlin/org/junit/jupiter/api/KotlinAssertionsTests.kt

+        val error = assertThrows<AssertionFailedError> {
+            assertInstanceOf<String>(null)
+        }


This feels meta, using a Kotlin assertion to test a Kotlin assertion, but it's probably ok, since they have different use cases for these two.

We have a few tests for assertThrows. Also, a similar approach is used for assertDoesNotThrow tests

junit-jupiter-engine/src/test/kotlin/org/junit/jupiter/api/KotlinAssertionsTests.kt

junit-jupiter-api/src/main/kotlin/org/junit/jupiter/api/Assertions.kt

TWiStErRob · 2023-08-27T15:42:34Z

junit-jupiter-api/src/main/kotlin/org/junit/jupiter/api/Assertions.kt

 fun assertAll(heading: String?, vararg executables: () -> Unit) =
    assertAll(heading, executables.toList().stream())


Reading this on GitHub makes me confused about the return type.
Again, not your code, but all your return types are nice and clean, probably worth aligning.

I've noticed that return type is not specified only for one-liners that return Unit. I assume, to resemble usual methods, where it isn't mandatory to specify Unit return type. For one-liners that return anything besides Unit, a return type is already specified

Issue: junit-team#1866

Invocation kind for the executable is UNKNOWN for assertTimeoutPreemptively methods Issue: junit-team#1866

JLLeitschuh · 2023-09-04T17:26:03Z

junit-jupiter-api/src/main/kotlin/org/junit/jupiter/api/Assertions.kt

@@ -134,6 +355,11 @@ inline fun <reified T : Throwable> assertThrows(message: String, executable: ()
 * @see Assertions.assertThrows
 */
 inline fun <reified T : Throwable> assertThrows(noinline message: () -> String, executable: () -> Unit): T {
+    contract {
+        callsInPlace(executable, UNKNOWN)


Why is this unknown?

See these two:

Add Kotlin contracts to exposed Kotlin API #3259 (comment)

Add Kotlin contracts to exposed Kotlin API #3259 (comment)

Contracts doesn't work well as being discussed in this conversation.
Perhaps, not specifying contracts for assertThrows and assertTimeoutPreemptively is a better option. For example, kotlin's runCatching method doesn't specify any contracts.
What do you think?

Contracts have been removed for all lambdas that may throw an exception

Would it be worth adding a comment into those methods that they intentionally don't have contracts and why?

Imo, adding comments regarding a contract absence everywhere adds nothing but noise. A person who's concerned by the missing contracts can check the commits and see why they aren't in place

Issue: junit-team#1866

awelless · 2023-09-23T13:22:45Z

Is there anything else that should be added/adjusted? Otherwise, I'd say the pr is in its final form

TWiStErRob

LGTM

TWiStErRob · 2023-09-23T15:57:53Z

junit-jupiter-engine/src/test/kotlin/org/junit/jupiter/api/KotlinAssertionsTests.kt

@@ -211,6 +211,59 @@ class KotlinAssertionsTests {
        assertMessageStartsWith(error, assertionMessage)
    }



My only note is that there are 18 methods affected, but only 6 new tests, and this may be normal because some contract stuff is not testable directly.

Added missing tests for assertion method overloads with message and message supplier

Issue: junit-team#1866

awelless · 2023-10-03T15:10:30Z

What do you think about the pr, @marcphilipp, @JLLeitschuh?

awelless · 2023-12-15T22:36:47Z

Hi. I believe the change is still relevant and the pr is ready to be reviewed.

marcphilipp · 2023-12-18T07:35:04Z

We'll consider it for 5.11, just need to find some time to review.

marcphilipp reviewed Apr 22, 2023

View reviewed changes

marcphilipp added the status: waiting-for-feedback label Apr 22, 2023

awelless requested a review from marcphilipp April 22, 2023 21:36

marcphilipp removed the status: waiting-for-feedback label Apr 23, 2023

marcphilipp reviewed Apr 23, 2023

View reviewed changes

junit-jupiter-api/src/main/kotlin/org/junit/jupiter/api/Assertions.kt Outdated Show resolved Hide resolved

awelless requested a review from marcphilipp April 23, 2023 19:06

JLLeitschuh reviewed May 26, 2023

View reviewed changes

TWiStErRob reviewed May 26, 2023

View reviewed changes

awelless requested review from JLLeitschuh and TWiStErRob May 26, 2023 20:15

TWiStErRob reviewed Jun 3, 2023

View reviewed changes

junit-jupiter-api/src/main/kotlin/org/junit/jupiter/api/Assertions.kt Outdated Show resolved Hide resolved

awelless requested a review from TWiStErRob June 3, 2023 21:39

awelless added 5 commits August 10, 2023 09:09

Merge branch 'main' into main

edbe73f

Mark new kotlin assetion methods as experimental

252cad2

Kotlin assertNull and assertNotNull methods are marked as experimental. Issue: junit-team#1866

Adjust assertThrows contract

1d5811b

Lambda invoked in assertThrows is marked as being called UNKNOWN number of times, since contracts do nothing with exception suppression Issue: junit-team#1866

Add contract for fail

19922d6

Created another fail method with a non-nullable lambda parameter Issue: junit-team#1866

awelless force-pushed the main branch from ed02ba8 to 19922d6 Compare August 10, 2023 07:12

TWiStErRob approved these changes Aug 10, 2023

View reviewed changes

awelless added 3 commits August 25, 2023 14:51

Don't return a value from assertNotNull

eccb259

Because of kotlin smart casts, there is no need to return a non-nullable value from `assertNotNull` methods Issue: junit-team#1866

Increase API version

e427978

Api version is increased to 5.11 for `assertNull` and `assertNotNull` methods Issue: junit-team#1866

Add assertInstanceOf methods

544b4e1

Added `assertInstanceOf` assertions with upcasting contracts Issue: junit-team#1866

TWiStErRob reviewed Aug 25, 2023

View reviewed changes

junit-jupiter-api/src/main/kotlin/org/junit/jupiter/api/Assertions.kt Outdated Show resolved Hide resolved

awelless added 3 commits August 25, 2023 16:31

assertInstanceOf methods accept nullable values

437d29e

Issue: junit-team#1866

Test for assertInstanceOf nullable upcast

43c08ea

Issue: junit-team#1866

Specify lambda InvocationKind explicitly for assertThrows methods

ff9fd39

Issue: junit-team#1866

awelless requested a review from TWiStErRob August 27, 2023 14:36

TWiStErRob reviewed Aug 27, 2023

View reviewed changes

Rename variable names in new assertion examples

5a70965

Issue: junit-team#1866

awelless added 3 commits September 1, 2023 22:22

Add API annotation for new fail assertion

666f73f

Issue: junit-team#1866

Make the comments be sentences

9920cf9

Issue: junit-team#1866

Adjust contract for assertTimeoutPreemptively

4d21486

Invocation kind for the executable is UNKNOWN for assertTimeoutPreemptively methods Issue: junit-team#1866

JLLeitschuh reviewed Sep 4, 2023

View reviewed changes

awelless requested a review from JLLeitschuh September 4, 2023 20:37

Removed contracts for methods where lambdas may throw exceptions

30b9b78

Issue: junit-team#1866

TWiStErRob approved these changes Sep 23, 2023

View reviewed changes

Add more tests for methods with contracts

41f00c0

Issue: junit-team#1866

Merge branch 'main' into main

69e9c38

sbrannen changed the title ~~Added Kotlin Contracts to Exposed Kotlin API~~ Add Kotlin contracts to exposed Kotlin API Dec 16, 2023

	private static <T extends Throwable> T assertThrows(Class<T> expectedType, Executable executable,
	Object messageOrSupplier) {

	try {
	executable.execute();
	}
	catch (Throwable actualException) {
	if (expectedType.isInstance(actualException)) {
	return (T) actualException;
	}
	else {
	UnrecoverableExceptions.rethrowIfUnrecoverable(actualException);
	throw assertionFailure() //
	.message(messageOrSupplier) //
	.expected(expectedType) //
	.actual(actualException.getClass()) //
	.reason("Unexpected exception type thrown") //
	.cause(actualException) //
	.build();
	}
	}
	throw assertionFailure() //
	.message(messageOrSupplier) //
	.reason(format("Expected %s to be thrown, but nothing was thrown.", getCanonicalName(expectedType))) //
	.build();
	}

		fun assertAll(heading: String?, vararg executables: () -> Unit) =
		assertAll(heading, executables.toList().stream())

		@@ -211,6 +211,59 @@ class KotlinAssertionsTests {
		assertMessageStartsWith(error, assertionMessage)
		}

Add Kotlin contracts to exposed Kotlin API #3259

Are you sure you want to change the base?

Add Kotlin contracts to exposed Kotlin API #3259

Conversation

awelless commented Apr 17, 2023 • edited

Overview

Definition of Done

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TWiStErRob May 26, 2023 • edited

Choose a reason for hiding this comment

awelless May 27, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TWiStErRob May 26, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

JLLeitschuh May 26, 2023 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TWiStErRob left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

TWiStErRob left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

awelless commented Sep 23, 2023

TWiStErRob left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

awelless commented Oct 3, 2023

awelless commented Dec 15, 2023

marcphilipp commented Dec 18, 2023

awelless commented Apr 17, 2023 •

edited

TWiStErRob May 26, 2023 •

edited

awelless May 27, 2023 •

edited

TWiStErRob May 26, 2023 •

edited

JLLeitschuh May 26, 2023 •

edited