Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Body token not expected error when trying to upload a large multipart file #29227

Closed
karolkrasnowski opened this issue Sep 30, 2022 · 2 comments
Assignees
Labels
type: bug A general bug type: regression A bug that is also a regression
Milestone

Comments

@karolkrasnowski
Copy link

karolkrasnowski commented Sep 30, 2022

I observe not an entirely deterministic issue while uploading big multipart files. It does not occur for small files. This error appears in the server logs:

java.lang.IllegalStateException: Body token not expected
at org.springframework.http.codec.multipart.PartGenerator$WritingFileState.body(PartGenerator.java:744) ~[spring-web-5.3.23.jar:5.3.23]

I created a minimal reproducible example to illustrate this issue.

This is the endpoint (full source code available here):

@RestController
class FilesController {

    @PostMapping(value = "/files")
    Mono<Integer> uploadFiles(@RequestBody Flux<Part> parts) {
        return parts
                .filter(FilePart.class::isInstance)
                .map(FilePart.class::cast)
                .flatMap(part -> DataBufferUtils.join(part.content())
                        .map(buffer -> {
                            byte[] data = new byte[buffer.readableByteCount()];
                            buffer.read(data);
                            DataBufferUtils.release(buffer);
                            return data;
                        })
                )
                .map(bytes -> bytes.length)
                .reduce(0, Integer::sum);
    }
}

And these are the tests (full source code available here):

@SpringBootTest
class BigMultipartFileUploadTest {

    private static final int NUMBER_OF_RESOURCES = 30;
    private WebTestClient webTestClient;

    @BeforeEach
    void before(ApplicationContext applicationContext) {
        webTestClient = WebTestClient.bindToApplicationContext(applicationContext).build();
    }

    /**
     * This test passes all the time
     */
    @Test
    void shouldUploadSmallFile() {
        // given
        int resourceSize = 5_000;
        var body = buildMultiValueMap(resourceSize);

        // expect
        sendPostRequest(body).isEqualTo(resourceSize * NUMBER_OF_RESOURCES);
    }

    /**
     * This test fails most of the time
     */
    @Test
    void shouldUploadBigFile() {
        // given
        int resourceSize = 5_000_000;
        var body = buildMultiValueMap(resourceSize);

        // expect
        sendPostRequest(body).isEqualTo(resourceSize * NUMBER_OF_RESOURCES);
    }

    private WebTestClient.BodySpec<Integer, ?> sendPostRequest(MultiValueMap<String, HttpEntity<?>> body) {
        return webTestClient.post()
                .uri("/files")
                .contentType(MediaType.MULTIPART_FORM_DATA)
                .body(BodyInserters.fromMultipartData(body))
                .exchange()
                .expectBody(Integer.class);
    }

    private MultiValueMap<String, HttpEntity<?>> buildMultiValueMap(int resourceSize) {
        MultipartBodyBuilder builder = new MultipartBodyBuilder();
        for (int i = 0; i < NUMBER_OF_RESOURCES; i++) {
            builder.part("item-" + i, newByteArrayResource(i, resourceSize));
        }
        return builder.build();
    }

    private ByteArrayResource newByteArrayResource(int fileIndex, int resourceSize) {
        return new ByteArrayResource(RandomUtils.nextBytes(resourceSize)) {
            @Override
            public String getFilename() {
                return "filename-" + fileIndex;
            }
        };
    }
}

As you can see, the only difference between these two tests is the resourceSize.
The first test always works, and the second one doesn't work most of the time.

Interestingly, this problem does not occur in the Spring Boot version 2.3.12.RELEASE, so it looks like a regression.

@spring-projects-issues spring-projects-issues added the status: waiting-for-triage An issue we've not yet triaged or decided on label Sep 30, 2022
@karolkrasnowski
Copy link
Author

I don't want this to sound rude, but when could I expect someone to take care of this? If this cannot be resolved in the foreseeable future, we will have to schedule a downgrade, so I'd like to know what we should prepare for.

@poutsma poutsma self-assigned this Oct 20, 2022
@poutsma poutsma added this to the 5.3.24 milestone Oct 20, 2022
@poutsma poutsma added type: bug A general bug type: regression A bug that is also a regression and removed status: waiting-for-triage An issue we've not yet triaged or decided on labels Oct 20, 2022
@karolkrasnowski
Copy link
Author

@poutsma thanks a lot for fixing this!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type: bug A general bug type: regression A bug that is also a regression
Projects
None yet
Development

No branches or pull requests

3 participants