Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[2.2] Flush buffer in streaming interface before writing zip data #1163

Merged
merged 1 commit into from Nov 9, 2022

Commits on Nov 9, 2022

  1. Flush buffer in streaming interface before writing zip data

    We ran into a really obscure issue when working on ansible/receptor#683.
    
    I'll try to make this at least somewhat digestable.
    
    Due to a bug in Kubernetes, AWX can't currently run jobs longer than 4 hours when deployed into Kubernetes. More context on that in ansible/awx#11805
    
    To address this issue, we needed a way to restart from a certain point in the logs. The only mechanism Kubernetes provides to do this is by passing "sinceTime" to the API endpoint for retrieving logs from a pod.
    
    Our patch in ansible/receptor#683 worked when we ran it locally, but in OpenShift, jobs errored when unpacking the zip stream at the end of the results of "ansible-runner worker". Upon further investigation this was because the timestamps of the last 2 lines were exactly the same:
    
    ```
    2022-11-09T00:07:46.851687621Z {"status": "successful", "runner_ident": "1"}
    2022-11-08T23:07:58.648753832Z {"zipfile": 1330}
    2022-11-08T23:07:58.648753832Z UEsDBBQAAAAIAPy4aFVGnUFkqQMAAIwK....
    ```
    
    After squinting at this code for a bit I noticed that we weren't flushing the buffer here like we do in the event_handler and other callbacks that are fired in streaming.py. The end. Ugh.
    shanemcd committed Nov 9, 2022
    Configuration menu
    Copy the full SHA
    64c9465 View commit details
    Browse the repository at this point in the history