Passing UploadFile objects into a StreamingResponse closes it in v0.106.0 but not v0.105.0

[Kludex]

Discussed in #10856

^{Originally posted by adrwz December 28, 2023}

First Check

• I added a very descriptive title here.
• I used the GitHub search to find a similar question and didn't find it.
• I searched the FastAPI documentation, with the integrated search.
• I already searched in Google "How to X in FastAPI" and didn't find any information.
• I already read and followed all the tutorial in the docs and didn't find an answer.
• I already checked if it is not related to FastAPI but to Pydantic.
• I already checked if it is not related to FastAPI but to Swagger UI.
• I already checked if it is not related to FastAPI but to ReDoc.

Commit to Help

• I commit to help with one of those options 👆

Example Code

async def upload_file_event(files: List[UploadFile]):
    print("2", files[0].filename, files[0].file._file.closed)

    # ... more code ...


@router.post("/upload_session_file")
async def upload_file(files: List[UploadFile] = File(...)):
    print("1", files[0].filename, files[0].file._file.closed)

    try:
        return StreamingResponse(
            upload_file_event(files),
            media_type="text/event-stream",
        )
    except Exception as error:
        logging.error(error)
        raise HTTPException(status_code=500, detail="Internal Server Error")

Description

Hitting this endpoint with an uploaded file on v0.105.0 will print:

1 filename false
2 filename false

Hitting this endpoint with an uploaded file on v0.106.0 will print:

1 filename false
2 filename true

Unfortunately this means I'm unable to upload files in a StreamingResponse with fastapi>=0.106.0

Operating System

macOS

Operating System Details

No response

FastAPI Version

0.105.0

Pydantic Version

2.5.3

Python Version

3.11.4

Additional Context

No response

[inverse-panda]

Is there any update on this issue? I have the same problem , just migrated from v0.103.0 to v0.108.0 and realized my tests are failing due the same issue. Is is related to this breaking change?
If yes, what is the proposed approach for handling the UploadFile in the background task?
my old practice is written below

async def upload_csv_to_gcloud_blob_storage(
    self, file: UploadFile, bucket_name: str, blob_storage_path:str
):
    """write the data to a bucket with"""
    storage_client = storage.Client()
    csv_bucket = storage_client.bucket(bucket_name)
    blob = csv_bucket.blob(blob_storage_path)

    try:
        with blob.open("wb") as buffer:
            shutil.copyfileobj(file.file, buffer)
    finally:
        file.file.close()
            
@app.post( "/csv_tables" )
async def upload_csv(
    csv_file: Annotated[UploadFile, File()],
    background_tasks: BackgroundTasks,
):
  
    background_tasks.add_task(
        upload_csv_to_gcloud_blob_storage,
        csv_file=csv_file,
        bucket_name="csv",
        blob_storage_path="csv_path",
    )
    return "OK"

[daviddavis]

We ran into this issue. Calling file.read() in code our began raising an error after upgrading fastapi: ValueError: I/O operation on closed file.

[adlmtl]

I'm having similar issue with a tempfile I'm yielding....

def create_temp_file():
    fd, path = tempfile.mkstemp(suffix='.csv')
    try:
        yield fd, path
    finally:
        os.unlink(path)

When I try to return the file in a FileResponse I hit:
RuntimeError(f"File at path {self.path} does not exist.")

[msehnout]

Is there any update on this issue? I have the same problem , just migrated from v0.103.0 to v0.108.0 and realized my tests are failing due the same issue. Is is related to this breaking change? If yes, what is the proposed approach for handling the UploadFile in the background task? my old practice is written below

It seems to be, removing this line prevent the file in the body from being closed:

fastapi/fastapi/routing.py

Line 231 in 958425a

async_exit_stack.push_async_callback(body.close)

It was introduced in the only commit changing the code between versions 105 and 106:
a4aa79e#diff-4d573079004a9f3d148baa4658e68e82b8a3d1a95d603fee8177daa92cf65c93R231

Unfortunately the get_request_handler is not trivial, so I cannot create PR without a bit of help.

[msehnout]

I went through the get_request_handler function again and maybe the behavior is intentional? I don't know. Anyway the problem is that it simply runs the endpoint handler:

fastapi/fastapi/routing.py

Line 294 in 958425a

raw_response = await run_endpoint_function(

and then closes the file. But in case the handler is asynchronous, like a background task:

@app.post(...)
async def foo():
   background_tasks.add_task(process_recording_file, file=file, ...)

it closes the file before the task can actually run. So I created a deep copy like this:

-background_tasks.add_task(process_recording_file, file=file, ...)
+background_tasks.add_task(process_recording_file, file=copy.deepcopy(file), ...)

and I close the file myself in the function:

async def process_recording_file(file: UploadFile...):
    try:
        ...
    finally:
        await file.close()

which fixes the issue even with the newest version of FastAPI. It is probably not very efficient given that the file is few MB in my case.

So maybe a better question would be, what is the expected usage of UploadFile combined with BackgroundTasks? https://fastapi.tiangolo.com/tutorial/background-tasks/

[wu-clan]

Hi， @msehnout

Interesting thinking, but I don't think it's a backstage task. Look at this.：encode/starlette#2176 (comment)

[Kludex]

How is my comment related?

[wu-clan]

I just want to express that the idea of @ msehnout may be backward incompatible because he mentioned background tasks，it seems to have nothing to do with your comments?

[rushilsrivastava]

I'm having similar issue with a tempfile I'm yielding....

def create_temp_file():
    fd, path = tempfile.mkstemp(suffix='.csv')
    try:
        yield fd, path
    finally:
        os.unlink(path)

When I try to return the file in a FileResponse I hit: RuntimeError(f"File at path {self.path} does not exist.")

Related discussion: #10850

I am assuming create_temp_file is used as a dependency. This error is happening because the dependency is fully resolved before the response is returned to the client. Previously, this worked because the exit code was executed after the response was sent. At the moment, I think the only way to work around this is using Background Tasks, as discussed in #2512.

[msehnout]

I'm not sure the problem I'm experiencing is actually an issue or a bad design choice on my side, so I created a discussion instead: #10936

[tjbck]

Any updates on this? deepcopying isn't an option for me.

[uniartisan]

Looking forward to an update. copy.deepcopy not work for me. I have no choice but to downgrade fastapi

[chrisk314]

I'm also hitting this. Need to perform some long running processing steps on uploaded files and it's no longer possible with BackgroundTasks due to the file being closed when returning the response. So this promise from the FastAPI docs on BackgroundTasks is now broken.

For example, let's say you receive a file that must go through a slow process, you can return a response of "Accepted" (HTTP 202) and process it in the background.

[chrisk314]

In case it helps others, for now I'm working around this issue by adding my own UploadFile class which wraps and patches the fastapi.UploadFile class. Tested and it's working as expected. BackgroundTask now processes data (i.e. uploads to S3) correctly in the background. Don't forget, using this approach you need to close the UploadFile manually in your BackgroundTask or otherwise. Not ideal or a long-term solution but it's doing the job.

from fastapi import UploadFile as _UploadFile

class UploadFile(_UploadFile):
    """Patches `fastapi.UploadFile` due to buffer close issue.

    See the related github issue:
    https://github.com/tiangolo/fastapi/issues/10857
    """

    def __init__(self, upload_file: _UploadFile) -> None:
        """Wraps and mutates input `fastapi.UploadFile`.

        Swaps `close` method on the input instance so it's a no-op when called
        by the framework. Adds `close` method of input as `_close` here, to be
        called later with overridden `close` method.
        """
        self.filename = upload_file.filename
        self.file = upload_file.file
        self.size = upload_file.size
        self.headers = upload_file.headers

        _close = upload_file.close
        setattr(upload_file, "close", self._close)
        setattr(self, "_close", _close)

    async def _close(self) -> None:
        pass

    async def close(self) -> None:  # noqa: D102
        await self._close()

[maxupp]

Also running into this, any plans for a timely fix?