the middleware can't be installed for the actual FastAPI version #214

alexted · 2023-02-12T15:28:33Z

Hi! Since FastAPI version 0.91.0, I started getting the following error when trying to run the application:

INFO:     Will watch for changes in these directories: ['/home/my_name/PycharmProjects/my-app']
INFO:     Uvicorn running on http://127.0.0.1:5000 (Press CTRL+C to quit)
INFO:     Started reloader process [15443] using WatchFiles
INFO:     Started server process [15472]
INFO:     Waiting for application startup.
ERROR:    Traceback (most recent call last):
  File "/home/my_name/.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/routing.py", line 671, in lifespan
    async with self.lifespan_context(app):
  File "/home/my_name/.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/routing.py", line 566, in __aenter__
    await self._router.startup()
  File "/home/my_name/.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/routing.py", line 648, in startup
    await handler()
  File "/home/my_name/PycharmProjects/my-app/core/app/service.py", line 75, in integrate_monitoring_plugin
    Instrumentator(excluded_handlers=["/health", "/metrics"]).instrument(app).expose(app, include_in_schema=False)
  File "/home/my_name/.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/prometheus_fastapi_instrumentator/instrumentation.py", line 121, in instrument
    app.add_middleware(
  File "/home/my_name/.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/applications.py", line 135, in add_middleware
    raise RuntimeError("Cannot add middleware after an application has started")
RuntimeError: Cannot add middleware after an application has started

ERROR:    Application startup failed. Exiting.

I initiate the application this way:

def create_app() -> FastAPI:
    app = FastAPI()

    #blah-blah-blah

    @app.on_event("startup")
    async def integrate_monitoring_plugin():
        Instrumentator(excluded_handlers=["/health", "/metrics"]).instrument(app).expose(app, include_in_schema=False)

    return app

Dencrash · 2023-02-14T12:00:38Z

We have the same problem with both the fastapi versions 0.91.0 and 0.92.0 being effected. It seems like adding a middleware after startup is not valid anymore, though they also fixed the fact that middlewares are initialized multiple times. Is there an alternative way to integrate the instrumentator?

duffn · 2023-02-14T20:54:59Z

We're encountering the same issue. Here's where this was introduced in Starlette. encode/starlette@51c1de1#diff-3eb07c0eea72d98f1b844e07844c2a68b7c11c63956b67d5afdd9c7f79ab946fR135

harochau · 2023-02-15T16:11:53Z

I have the same problem.
Maybe we can use https://github.com/stephenhillier/starlette_exporter instead? Does anybody have experience with that library, is it a drop-in replacement?

alexted · 2023-02-15T16:19:34Z

Yes, I too am thinking of switching to another library.

alfaro28 · 2023-02-16T02:34:05Z

Is there any reason to implement the instrumentator on the "startup" event? Just moving the instrumentator outside of the event makes things work

alexted · 2023-02-16T04:15:44Z

This works until you run unit tests.

jonathanloske · 2023-02-21T08:16:52Z

Is there any reason to implement the instrumentator on the "startup" event? Just moving the instrumentator outside of the event makes things work

According to the maintainer:

Prometheus expects a fully initialized ASGI client, where uvicorn now delays execution in newer versions. Closing.

alfaro28 · 2023-02-21T16:22:53Z

sorry I should have been more clear, what I'm currently doing is creating the instrumentator and "instrument" the app outside of the startup event, but actually "exposing" it on the startup event and seems to be working fine

duffn · 2023-02-21T19:21:02Z

sorry I should have been more clear, what I'm currently doing is creating the instrumentator and "instrument" the app outside of the startup event, but actually "exposing" it on the startup event and seems to be working fine

Can you share a bit of code here please?

alfaro28 · 2023-02-21T22:02:01Z

Sure,, just a quick note, I'm using start_http_server because I run my prometheus server on a diferent port, but I asume you could save the Instrumentator variable and just call .expose on the startup_event function

from fastapi import FastAPI
from prometheus_client import start_http_server
from prometheus_fastapi_instrumentator import Instrumentator

async def startup_event():
    start_http_server(8080)
    
app = FastAPI(
    on_startup=[startup_event]
)

Instrumentator(
    excluded_handlers=["/metrics"],
).instrument(app)

harochau · 2023-02-23T12:42:16Z

can confirm, this works with the new Fastapi

instrumentator = Instrumentator().instrument(app)
@app.on_event("startup")
async def _startup():
    instrumentator.expose(app, endpoint='/metrics', tags=['metrics'])

trallnag · 2023-02-25T22:56:40Z

Is there any reason to implement the instrumentator on the "startup" event? Just moving the instrumentator outside of the event makes things work

The README was updated to use startup around 6 months ago due to #80. It is probably best to update the README again with the next release. Just tried it (while testing other things) and for me it works without startup fine.

Here are relevant dependencies:

❯ pip list
Package                           Version Editable project location
--------------------------------- ------- ----------------------------
fastapi                           0.92.0
prometheus-client                 0.16.0
prometheus-fastapi-instrumentator 100.0.0
starlette                         0.25.0
uvicorn                           0.20.0

The code:

from fastapi import FastAPI

from prometheus_fastapi_instrumentator import Instrumentator

app = FastAPI()

@app.get("/app")
def read_main():
    return {"message": "Hello World from main app"}

Instrumentator().instrument(app).expose(app)

Reproducible examples would be great as I am not using the package actively anymore.

alexted · 2023-02-26T06:37:29Z

it breaks all my unit tests:

trallnag · 2023-02-26T16:04:07Z

@alexted does this also happen if you use https://github.com/stephenhillier/starlette_exporter? I think it is related to how the prometheus client library works. In the unit tests of this package here I call the following function at the beginning of every relevant unit test: https://github.com/trallnag/prometheus-fastapi-instrumentator/blob/master/tests/helpers/utils.py#L4

Another solution seems to be to use a new registry for every unit test: rycus86/prometheus_flask_exporter#17

Current version of prometheus-fastapi-instrumentator does not support passing custom registry, but this feature has already been merged as part of clean up and I will release it today

saschnet · 2023-02-26T16:21:07Z

Is there anything pointing against the solution by @alfaro28 / @harochau?

For me, it is working with all circumstances (unit tests + the app itself) and is much simpler than adding a new registry for every unit test etc.

trallnag · 2023-02-26T16:23:13Z

@saschnet, I don't know 😆 I will update the README and switch to the example provided by @harochau

alexted · 2023-02-28T07:04:52Z

@alexted does this also happen if you use https://github.com/stephenhillier/starlette_exporter?

No, everything works fine when using this library. The application runs and the unit tests pass.

alexted · 2023-03-04T06:31:35Z

@trallnag thanks for the release.
with the new version I started to get another error when running unit tests:

tests/v1_1/advertisers/test_get_expenses.py:107: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/httpx/_client.py:1757: in get
    return await self.request(
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/httpx/_client.py:1533: in request
    return await self.send(request, auth=auth, follow_redirects=follow_redirects)
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/httpx/_client.py:1620: in send
    response = await self._send_handling_auth(
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/httpx/_client.py:1648: in _send_handling_auth
    response = await self._send_handling_redirects(
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/httpx/_client.py:1685: in _send_handling_redirects
    response = await self._send_single_request(request)
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/httpx/_client.py:1722: in _send_single_request
    response = await transport.handle_async_request(request)
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/httpx/_transports/asgi.py:162: in handle_async_request
    await self.app(scope, receive, send)
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/fastapi/applications.py:271: in __call__
    await super().__call__(scope, receive, send)
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/applications.py:118: in __call__
    await self.middleware_stack(scope, receive, send)
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/middleware/errors.py:184: in __call__
    raise exc
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/middleware/errors.py:162: in __call__
    await self.app(scope, receive, _send)
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/middleware/cors.py:84: in __call__
    await self.app(scope, receive, send)
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/middleware/base.py:108: in __call__
    response = await self.dispatch_func(request, call_next)
src/utils/middlewares/log_requests.py:19: in log_requests
    response_body = [chunk async for chunk in response.body_iterator]
src/utils/middlewares/log_requests.py:19: in <listcomp>
    response_body = [chunk async for chunk in response.body_iterator]
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/middleware/base.py:98: in body_stream
    raise app_exc
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/middleware/base.py:70: in coro
    await self.app(scope, receive_or_disconnect, send_no_error)
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/middleware/base.py:109: in __call__
    await response(scope, receive, send)
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/responses.py:270: in __call__
    async with anyio.create_task_group() as task_group:
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/anyio/_backends/_asyncio.py:662: in __aexit__
    raise exceptions[0]
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/responses.py:273: in wrap
    await func()
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/middleware/base.py:134: in stream_response
    return await super().stream_response(send)
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/responses.py:262: in stream_response
    async for chunk in self.body_iterator:
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/middleware/base.py:98: in body_stream
    raise app_exc
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/middleware/base.py:70: in coro
    await self.app(scope, receive_or_disconnect, send_no_error)
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/middleware/base.py:109: in __call__
    await response(scope, receive, send)
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/responses.py:270: in __call__
    async with anyio.create_task_group() as task_group:
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/anyio/_backends/_asyncio.py:662: in __aexit__
    raise exceptions[0]
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/responses.py:273: in wrap
    await func()
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/middleware/base.py:134: in stream_response
    return await super().stream_response(send)
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/responses.py:262: in stream_response
    async for chunk in self.body_iterator:
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/middleware/base.py:98: in body_stream
    raise app_exc
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/starlette/middleware/base.py:70: in coro
    await self.app(scope, receive_or_disconnect, send_no_error)
../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/prometheus_fastapi_instrumentator/middleware.py:181: in __call__
    instrumentation(info)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

info = <prometheus_fastapi_instrumentator.metrics.Info object at 0x7feb23667fd0>

    def instrumentation(info: Info) -> None:
>       TOTAL.labels(info.method, info.modified_status, info.modified_handler).inc()
E       NameError: free variable 'TOTAL' referenced before assignment in enclosing scope

../../.cache/pypoetry/virtualenvs/my-app-PzElRY_T-py3.10/lib/python3.10/site-packages/prometheus_fastapi_instrumentator/metrics.py:625: NameError

AliSayyah · 2023-03-05T17:36:11Z

For now, I'm excluding 'prometheus_fastapi_instrumentator' when running tests.
I'm using 'pytest' so incase anyone having the same problem:

instrumentor = None
    if "pytest" not in sys.modules:
        instrumentor = PrometheusFastApiInstrumentator(
            should_group_status_codes=False
        ).instrument(
            app,
        )

and:

    @app.on_event("startup")
    async def _startup() -> None: 
        ...
        if "pytest" not in sys.modules:
            setup_prometheus(app, instrumentor)
        ...

note: this would only work if you haven't imported 'pytest' outside your tests. there may be cleaner ways to check this.

trallnag · 2023-03-05T17:46:53Z

@alexted, probably related to #219

trallnag · 2023-03-08T19:53:03Z

The NameError thing should be fixed now with the latest release https://github.com/trallnag/prometheus-fastapi-instrumentator/releases/tag/v5.11.0

AliSayyah · 2023-03-08T20:33:40Z

The NameError thing should be fixed now with the latest release https://github.com/trallnag/prometheus-fastapi-instrumentator/releases/tag/v5.11.0

Thanks. NameError is gone.

alexted · 2023-03-10T10:12:41Z

ty @trallnag

alexted changed the title ~~app doesn't start due to an exception in the new version of fastapi~~ app doesn't start due to an exception when using a actual version of FastAPI Feb 15, 2023

alexted changed the title ~~app doesn't start due to an exception when using a actual version of FastAPI~~ the middleware can't be installed for the actual FastAPI version Feb 15, 2023

refl3ct1on mentioned this issue Mar 13, 2023

Upgrading FastAPI version breaks the app s3rius/FastAPI-template#149

Open

alexted closed this as completed Mar 26, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

the middleware can't be installed for the actual FastAPI version #214

the middleware can't be installed for the actual FastAPI version #214

alexted commented Feb 12, 2023 •

edited

Dencrash commented Feb 14, 2023

duffn commented Feb 14, 2023

harochau commented Feb 15, 2023

alexted commented Feb 15, 2023

alfaro28 commented Feb 16, 2023

alexted commented Feb 16, 2023

jonathanloske commented Feb 21, 2023

alfaro28 commented Feb 21, 2023

duffn commented Feb 21, 2023

alfaro28 commented Feb 21, 2023

harochau commented Feb 23, 2023 •

edited

trallnag commented Feb 25, 2023

alexted commented Feb 26, 2023

trallnag commented Feb 26, 2023

saschnet commented Feb 26, 2023 •

edited

trallnag commented Feb 26, 2023

alexted commented Feb 28, 2023

alexted commented Mar 4, 2023 •

edited

AliSayyah commented Mar 5, 2023 •

edited

trallnag commented Mar 5, 2023

trallnag commented Mar 8, 2023

AliSayyah commented Mar 8, 2023

alexted commented Mar 10, 2023

the middleware can't be installed for the actual FastAPI version #214

the middleware can't be installed for the actual FastAPI version #214

Comments

alexted commented Feb 12, 2023 • edited

Dencrash commented Feb 14, 2023

duffn commented Feb 14, 2023

harochau commented Feb 15, 2023

alexted commented Feb 15, 2023

alfaro28 commented Feb 16, 2023

alexted commented Feb 16, 2023

jonathanloske commented Feb 21, 2023

alfaro28 commented Feb 21, 2023

duffn commented Feb 21, 2023

alfaro28 commented Feb 21, 2023

harochau commented Feb 23, 2023 • edited

trallnag commented Feb 25, 2023

alexted commented Feb 26, 2023

trallnag commented Feb 26, 2023

saschnet commented Feb 26, 2023 • edited

trallnag commented Feb 26, 2023

alexted commented Feb 28, 2023

alexted commented Mar 4, 2023 • edited

AliSayyah commented Mar 5, 2023 • edited

trallnag commented Mar 5, 2023

trallnag commented Mar 8, 2023

AliSayyah commented Mar 8, 2023

alexted commented Mar 10, 2023

alexted commented Feb 12, 2023 •

edited

harochau commented Feb 23, 2023 •

edited

saschnet commented Feb 26, 2023 •

edited

alexted commented Mar 4, 2023 •

edited

AliSayyah commented Mar 5, 2023 •

edited