trace_request_start does not include auto-generated headers

[hashstat]

🐞 Describe the bug

Tracing requests using the TraceConfig signals is essential for insight into what is sent over the wire, especially for encrypted https traffic. The trouble is that some of the data is incomplete. The request headers passed to trace_request_start handlers is missing auto-generated headers, making diagnosing failed requests more difficult.

💡 To Reproduce

1. Create a virtual environment
2. Install aiohttp, pytest, and pytest-aiohttp
3. Save the script below as test_auto_headers.py
4. Run the tests: pytest test_auto_headers.py

# test_auto_headers.py

import aiohttp
from aiohttp import web

async def test_trace_headers_match_auto_headers(aiohttp_client):
    request_headers: dict
    trace_headers: dict

    async def local_request(request):
        nonlocal request_headers
        request_headers = dict(request.headers)
        return web.Response()

    app = web.Application()
    app.router.add_get('/local', local_request)

    async def on_request_start(session, context, params):
        nonlocal trace_headers
        trace_headers = dict(params.headers)

    trace_config = aiohttp.TraceConfig()
    trace_config.on_request_start.append(on_request_start)

    client = await aiohttp_client(app, trace_configs=[trace_config])

    data = {'request': 'add', 'id': 1234, 'value': 'Hello World!'}

    async with client.get('/local', json=data) as response:
        assert response.status == 200
    assert trace_headers == request_headers

💡 Expected behavior

The test should succeed, indicating that the headers passed to trace_request_start handlers are identical to those received by the test server.

📋 Logs/tracebacks

$ pytest test_auto_headers.py -vv
==================================================================== test session starts =====================================================================
platform linux -- Python 3.7.7, pytest-6.0.2, py-1.9.0, pluggy-0.13.1
cachedir: .pytest_cache
plugins: aiohttp-0.3.0
collected 1 item                                                                                                                                             

test_auto_headers.py::test_trace_headers_match_auto_headers[pyloop] FAILED                                                                             [100%]

========================================================================== FAILURES ==========================================================================
_______________________________________________________ test_trace_headers_match_auto_headers[pyloop] ________________________________________________________

aiohttp_client = <function aiohttp_client.<locals>.go at 0x7f8eee0724d0>

    async def test_trace_headers_match_auto_headers(aiohttp_client):
        request_headers: dict
        trace_headers: dict
    
        async def local_request(request):
            nonlocal request_headers
            request_headers = dict(request.headers)
            return web.Response()
    
        app = web.Application()
        app.router.add_get('/local', local_request)
    
        async def on_request_start(session, context, params):
            nonlocal trace_headers
            trace_headers = dict(params.headers)
    
        trace_config = aiohttp.TraceConfig()
        trace_config.on_request_start.append(on_request_start)
    
        client = await aiohttp_client(app, trace_configs=[trace_config])
    
        data = {'request': 'add', 'id': 1234, 'value': 'Hello World!'}
    
        async with client.get('/local', json=data) as response:
            assert response.status == 200
>       assert trace_headers == request_headers
E       AssertionError: assert {} == {'Accept': '*/*',\n 'Accept-Encoding': 'gzip, deflate',\n 'Content-Length': '55',\n 'Content-Type': 'application/json',\n 'Host': '127.0.0.1:40225',\n 'User-Agent': 'Python/3.7 aiohttp/3.7.2'}
E         Right contains 6 more items:
E         {'Accept': '*/*',
E          'Accept-Encoding': 'gzip, deflate',
E          'Content-Length': '55',
E          'Content-Type': 'application/json',
E          'Host': '127.0.0.1:40225',
E          'User-Agent': 'Python/3.7 aiohttp/3.7.2'}
E         Full diff:
E           {
E         +  ,
E         -  'Accept': '*/*',
E         -  'Accept-Encoding': 'gzip, deflate',
E         -  'Content-Length': '55',
E         -  'Content-Type': 'application/json',
E         -  'Host': '127.0.0.1:40225',
E         -  'User-Agent': 'Python/3.7 aiohttp/3.7.2',
E           }

test_auto_headers.py:29: AssertionError
================================================================== short test summary info ===================================================================
FAILED test_auto_headers.py::test_trace_headers_match_auto_headers[pyloop] - AssertionError: assert {} == {'Accept': '*/*',\n 'Accept-Encoding': 'gzip, def...
===================================================================== 1 failed in 0.09s ======================================================================

📋 Your version of the Python

3.7.7

📋 Your version of the aiohttp/yarl/multidict distributions

$ pip show aiohttp multidict yarl
Name: aiohttp
Version: 3.7.2
Summary: Async http client/server framework (asyncio)
Home-page: https://github.com/aio-libs/aiohttp
Author: Nikolay Kim
Author-email: fafhrd91@gmail.com
License: Apache 2
Requires: attrs, yarl, chardet, async-timeout, multidict, typing-extensions
Required-by: pytest-aiohttp, aioresponses, aiohttp-retry, idt
---
Name: multidict
Version: 4.7.6
Summary: multidict implementation
Home-page: https://github.com/aio-libs/multidict
Author: Andrew Svetlov
Author-email: andrew.svetlov@gmail.com
License: Apache 2
Requires: 
Required-by: yarl, aiohttp
---
Name: yarl
Version: 1.5.1
Summary: Yet another URL library
Home-page: https://github.com/aio-libs/yarl/
Author: Andrew Svetlov
Author-email: andrew.svetlov@gmail.com
License: Apache 2
Requires: multidict, typing-extensions, idna
Required-by: aiohttp

📋 Additional context

client

[derlih]

Looks similar to #5105

[hashstat]

@derlih It looks like that covers our use case. Any idea when v3.8 will be released?

[derlih]

@asvetlov Do you know when this header tracing API will be released?

[asvetlov]

aiohttp 3.8 will bee released soon. Closing the issue