Test case with container and lots of output eventually deadlock and likely cause test to fail or cause other strange issues with test code

[manuel-koch]

We have a testcase that runs multiple containers with docker-compose that create considerable output on console.

Somehow we noticed that sooner or later the test seems to run into a timeout because the applications in the container under test get stuck into deadlocks related to filled-up stdout buffers and simply don't output anything anymore and cease to function properly.

The root cause seems to be the call of the pytest-plugin ( equivalent to docker-compose up ) that doesn't read the output gathered by docker-compose and stdout/stderr buffers within containers eventually fill up, blocking any attempt to further output anything.

Here is a simple docker-compose setup and a test to show the scenario. I.e. it runs a simple busybox that outputs chunks of arbitrary strings while the pytest just runs simulating test-code for the container under test.

As stdout fills up within container, it eventually causes the output to stop, while the test is still running.

E.g. the python test looks like:

import time
import datetime

pytest_plugins = ["docker_compose"]

def test_lots_of_output(module_scoped_container_getter):
    while True:
        print(f"Test at {datetime.datetime.now()} still running...")
        time.sleep(5)
        # eventually there will be no more output from our busybox container
        # as it deadlocked because if output buffer exhaustion caused
        # by running `docker-compose up` without reading output gathered by `docker-compose up`...

docker-compose setup looks like:

version: "3.4"
services:
  busybox:
    image: busybox
    command: sh -c "while true ; do date ; cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 2048 | head -n 1 ; sleep 1 ; done"

A helper script to start the test ( it spawns docker-compose logs -f to show that container under test eventually gets stuck deadlocked ):

#!/bin/sh
( sleep 10 ; docker-compose logs -f --tail=0 ) &
pytest --capture=no --docker-compose-no-build

The only workaround for this problem that we came up is to start containers detached manually ( docker-compose up -d ) before starting pytest and executing the test functionality.

Shouldn't the pytest plugin read output of docker-compose up to prevent such filling up of buffers and eventually deadlock the containers under test ?

Used pytest-docker-compose 3.2.1

[askurihin]

Seems relevant #76