Can I recreate a new clientsession in session

[cmcai0104]

I want to concurrent request for one url like following:
`
async def fetch(data):
url = ''
async with session.post(url=url, json=data) as resp:
text = await resp.text()
return text

async def fetch_all(datas):
return await asyncio.gather(*[fetch(data) for data in datas])

loop = asyncio.get_event_loop()
datas = []
session = aiohttp.ClientSession()
results = loop.run_until_complete(fetch_all(datas))
loop.run_until_complete(session.close())
loop.run_until_complete(asyncio.sleep(3))
loop.close()
`

but I always meet 504 in request. so I want to recreate session in function fetch when status equal 504:

`
session = None

async def fetch(data):
global session
url = ''
async with session.post(url=url, json=data) as resp:
if resp.status == 504:
await session.close()
session = aiohttp.ClientSession()
return await fetch(data)
else:
text = await resp.text()
return text

async def fetch_all(datas):
return await asyncio.gather(*[fetch(data) for data in datas])

loop = asyncio.get_event_loop()
datas = []
session = aiohttp.ClientSession()
results = loop.run_until_complete(fetch_all(datas))
loop.run_until_complete(session.close())
loop.run_until_complete(asyncio.sleep(3))
loop.close()
`
Then I meet: Connector is closed.

aiohttp.client_exceptions.ClientConnectionError: Connector is closed.

[Dreamsorcerer]

The problem is surely that you close a session in 1 task while the other tasks are in the middle of making a request with that session.
I'd suggest not messing around with globals and low-level asyncio APIs.

If you're going to close a session in a fetch function, I'm not sure there's an obvious, clean way to replace the session like that. For this example, I'd probably just use a session per fetch.

async def fetch(data):
    async with ClientSession() as sess:
        async with sess.post(...):
            ...

async def main():
    return await asyncio.gather(...)

results = asyncio.run(main())

But, I'm not sure why creating a new session would help with a 504 error, that should just mean that the upstream server is timing out (maybe because it's under too much load).

[cmcai0104]

is there any method to aviod http request error like 504 in aiohttp

[Dreamsorcerer]

Like I said, it indicates the upstream server is timing out. It's an issue with the server, not aiohttp.

The only thing you can try, is to slow down the requests so it doesn't overload the server.
Maybe something along the lines of:

async def fetch(data, sess, sem):
    async with sem:
        async with sess.post(...):
            ...

async def main(data_seq):
    sem = asyncio.Semaphore(10)
    async with ClientSession() as sess:
        return await asyncio.gather(*(fetch(d, sess, sem) for d in data_seq))

results = asyncio.run(main([...]))

Which should limit it to 10 requests in parallel.