Error 'NoneType' object has no attribute 'getall' in response processing in aiohttp in production application

[kayforkaran]

Hi team,
we are using aiohttp to call microservices in our organisation with the following spec
aiohttp version 3.8.1
python version 3.10.7
We started getting a large number of errors with the message 'NoneType' object has no attribute 'getall' from 17th Jan 2023 (we got a few before that as well but starting 17th we saw significant increase in them). I am not able to figure out what may have caused the issue because there were no version bumps or new API calls that were added during this time.

From the trace the line where the error originates is -

aiohttp/aiohttp/client_reqrep.py

Line 878 in 74340cb

for hdr in self.headers.getall(hdrs.SET_COOKIE, ()):

The self.headers is None which is why the error.

Attaching the complete error trace trace.txt with truncated application code here.
Would appreciate any leads or pointers to the cause of the issue and suggestions for possible solves.

[Dreamsorcerer]

Well, that attribute is set 2 lines earlier:
https://github.com/aio-libs/aiohttp/blob/3.8/aiohttp/client_reqrep.py#L925

Tracing back through the code, it appears that the headers are being returned at this point:

aiohttp/aiohttp/http_parser.py

Line 504 in ed04b4d

return (headers, raw_headers, close_conn, encoding, upgrade, chunked)

And a few lines before the return there is a headers.get() call, so there's no way it can be a None object at that point. I can't see anything else while tracing back to that return statement which could result in headers being changed to None.

My only guess is that you could be using a subclass of one of those objects which returns a different value, or otherwise there is some dodgy code that is just setting the headers to None somewhere else. But, without a reproducer to debug, I can't guess better than that.

Noting the newrelic in your traceback, I wonder if you should be filing a bug there. It certainly looks like they are doing some suspicious things with private _headers variables (though I'm not seeing anything that specifically looks like the problem): https://github.com/newrelic/newrelic-python-agent/blob/main/newrelic/hooks/framework_aiohttp.py#L67

[Dreamsorcerer]

Though, if you can remove newrelic and see if the errors disappear, that'd certainly figure out if it was a problem with their code or not.

[AlexFrundin]

Hello!
Are there any solutions to this problem?
I have the same problem - Exception (for hdr in self.headers.getall(hdrs.SET_COOKIE, ()): AttributeError: 'NoneType' object has no attribute 'getall')
Using

• python 3.12
• aiohttp 3.7.0
• fastapi 0.95.2
• uvicorn (anyio backend asyncio)
• viztracer as asgi middleware (also checked others tracer/profiler)

With tracer enabled
If in the fastapi request handler, an additional aiohttp request is made by the client, a similar situation occurs.

Abstract example

@app.get('')
def test_request():
   client = MyClientAiohttp()
   async with client.get('url') as resp:
       resp_text = await resp.text()

Tried various implementations of aiohttp client.

P.S. When disable logging of function arguments and logging of function return values in the tracer settings, everything works

[AlexFrundin]

file client.py row 557

try:
      try:
          resp = await req.send(conn)  # after resp.headers is None (Prior to this call, headers exist)
          if resp.headers is None:
                  client_logger.info(resp._writer._state) # Pending
          try:
              # log.info(f'!!!!!!!AIOHTTP TRY START  {conn}')
              await resp.start(conn)

await writer.write_headers(status_line, self.headers)
self._writer = self.loop.create_task(self.write_bytes(writer, conn)) 

If, with the tracer disabled, added logger to check states

resp = await req.send(conn)
client_logger.info(resp._writer._state) 

I get
Exception (for hdr in self.headers.getall(hdrs.SET_COOKIE, ()): AttributeError: 'NoneType' object has no attribute 'getall')

[AlexFrundin]

it's ok.

 self._headers = message.headers  # type is CIMultiDictProxy
 self._raw_headers = message.raw_headers  # type is Tuple[bytes, bytes]

But if you look at the code, these lines do not affect self.headers

As I understand it, using the writer encapsulates the value of self.headers.
If after receiving resp = await req.send(conn) I try to force the value of resp.headers - I get a readOnly error

[webknjaz]

Why are you using aiohttp 3.7.0, again? I don't think we need to bother about it, as code there might differ from 3.9/3.8.

[AlexFrundin]

I((feel stupid(((
my version aiohttp - 3.8.2
Now I will check with version 3.9

[Dreamsorcerer]

But if you look at the code, these lines do not affect self.headers

Yes they do. It's a property method which simply returns self._headers.

[GDYendell]

I have also just had this issue using version 3.8.5 and Python 3.11.4.

I was stepping through in a debugger to solve another issue and when I fixed that, I hit this error. I seem to be able to reliably reproduce it. If I put a breakpoint on the line:

self._headers = message.headers  # type is CIMultiDictProxy

and then step over, self.headers remains as None and it crashes on continue, but if I put a breakpoint on the line after:

self._raw_headers = message.raw_headers  # type is Tuple[bytes, bytes]

then self.headers is correctly populated and it runs happily on continue. It also doesn't happen when running normally or without any breakpoints.

I doubt anyone else can reproduce this, as it seems too strange to be true and I feel like I must be misunderstanding something, but if you can maybe this is a clue.