When returning a ResponseEntity with a Flux while the function is suspended, it fails to encode the body

[emilwihlander]

Affects: 5.3.13

When returning a ResponseEntity<Flux<[some class]>> when the function is suspended it unnests to Flux<[some class]> (tested: 5.3.13) while it previously unnested to [some class] (tested: 5.3.11).

Example

@RestController
class Controller(val service: SomeService) {
  @GetMapping("/")
  suspend fun getStrings(): ResponseEntity<Flux<ByteBuffer>> {
    val strs: Flux<String> = service.getStrings()
    // some coroutine function is called
    val headers = HttpHeaders()
    // Set headers
    return ResponseEntity.ok().headers(headers).body(strs)
  }
}

/**
 * <= 5.3.11
 * This worked fine
 * 
 * 5.3.13
 * Throws:
 * org.springframework.http.converter.HttpMessageNotWritableException: No Encoder for [reactor.core.publisher.Flux<java.nio.ByteBuffer>] with preset Content-Type 'application/octet-stream'
 **/

This is due to the change in org.springframework.web.reactive.result.method.annotation.ResponseEntityResultHandler, where the function handleResult was changed to handle coroutinescommit. It runs nested once instead of twice, as it did before.

Mixing reactor and coroutines is bad practice, I'd guess. Therefore, I'm not certain what the desired behaviour is. Any thoughts?

[rrittwag]

This small project provides an example. It passes a Flux<DataBuffer> from GridFS to ResponseEntity.body.

The unit test works when using Spring Boot 2.5.5 (Spring Framework 5.3.11), but fails on above versions.