Webserver crashes after few API calls from external server after upgrade to 0.15.8

[aballiet]

Describe the issue

Webserver was working fine in 0.14.4, after upgrading to 0.15.8, webserver systematically crashes after few API calls (2 calls).

We tried a bunch of things and noticed the server struggles globally.

Here is an example of a flow which created a lot of scaling events on Kestra server and can make it crash :

id: kestra-webserver-crasher
namespace: dev
tasks:
- id: spike-webserver-memory
  type: io.kestra.plugin.scripts.python.Script
  runner: PROCESS
  beforeCommands:
  - "pip install requests > /dev/null"
  warningOnStdErr: false
  docker:
    pullPolicy: ALWAYS
  script: |
    import requests

    BASE_SERVICE_URL = "http://kestra-service.kestra:8080/"

    # Get all running excections
    search = {"size":20}
    response = requests.get(f"{BASE_SERVICE_URL}/api/v1/executions/search", params=search)
    total = response.json()["total"]
    if total == 0:
        print("No executions currently running")
    else:
        print(f"{total} executions to fetch logs for")
        executions = [
            {"id": result["id"], "flow": result["flowId"], "namespace": result["namespace"]}
            for result in response.json()["results"]
        ]
        for execution in executions:
            id = execution["id"]
            log_search = {"q": ""}
            logs = requests.get(f"{BASE_SERVICE_URL}/api/v1/logs/{id}", params=log_search)
            print(logs)

Environment

• Kestra Version: 0.15.8 (also tested with 0.15.9, same issue)
• Operating System (OS/Docker/Kubernetes): Kubernetes
• Java Version (if you don't run kestra in Docker):

[loicmathieu]

Hi,
Do you have some logs on the Kestra webserver side?

[aballiet]

updated the issue @loicmathieu

[loicmathieu]

There is a workaround with the following configuration:

  micronaut:
    server:
      max-request-size: 1GB
      netty:
        server-type: full_content

But it would limit the request size to 1GB and silently ignore bigger requests, this can be an issue if you start a flow with total inputs size of more than 1GB.

We're still working with the Micronaut team to find a proper fix, see micronaut-projects/micronaut-core#10677

[tchiotludo]

@aballiet do you still have the issues?