how to configure a proxy?

[juanmol]

hi!, i have test the example app without problems in a server with direct connection to internet (without proxy). I need to get it working on other server with proxy. In this server, if i do a wget to the endpoint, i can see the traffic throught the proxy, but if i run the python script, i can't see anything on the proxy server.
I had test in other server with a diferent proxy, and can't get it working again. How can i configure proxy for this? I have this messages:
awscrt.exceptions.AwsCrtError: AwsCrtError(name='AWS_IO_SOCKET_TIMEOUT', message='socket operation timed out.', code=1048)
or
awscrt.exceptions.AwsCrtError: AwsCrtError(name='AWS_IO_SOCKET_NO_ROUTE_TO_HOST', message='socket connect failure, no route to host.', code=1049)
For enviroment test, i had configurated http_proxy and http_proxy

[joguSD]

I believe some provisional proxy support is available upstream in the AWS CRT but we'll have to expose and forward that configuration within the context of this library or read from the env vars. Marking as a feature request.

[nateprewitt]

Hi @juanmol, the AWS CRT has added provisional support for HTTP_PROXY and HTTPS_PROXY environment variables (awslabs/aws-crt-python#313) in awscrt==0.12.3.

If you get a moment to check this out, we'd be interested to know if this solves your issues. Thanks!

[YichiRockyZhang]

Hi, I'm trying to use my company's internal proxy, and it doesn't seem like those environment variables are being picked up. I'm using Windows (aws_transcribe=0.5.2, awscrt=0.13.5), and have the following environment variables set:

HTTP_PROXY=http://proxy.companyname.net:10443, HTTPS_PROXY=http://proxy.companyname.net:10443, NO_PROXY="localhost,127.0.0.1,.companyname.net,.companyname.com", AWS_PROFILE="C:\Users\...\.aws", AWS_DEFAULT_PROFILE="[only profile availible]".

When I run the code here, I get

Traceback (most recent call last):
  File ".../test-transcription.py", line 63, in <module>
    loop.run_until_complete(basic_transcribe())
  File "C:\...\base_events.py", line 646, in run_until_complete
    return future.result()
  File "...\test-transcription.py", line 40, in basic_transcribe
    stream = await client.start_stream_transcription(
  File "...\amazon_transcribe\client.py", line 170, in start_stream_transcription
    response = await session.make_request(
  File "...\amazon_transcribe\httpsession.py", line 235, in make_request
    connection = await self._get_connection(parsed_url)
  File "...\amazon_transcribe\httpsession.py", line 202, in _get_connection
    connection = await self._create_connection(parsed_url)
  File "...\amazon_transcribe\httpsession.py", line 178, in _create_connection
    connection = await asyncio.wrap_future(connect_future)
awscrt.exceptions.AwsCrtError: AWS_IO_SOCKET_TIMEOUT: socket operation timed out.

I've tried modifying httpsession.py to pass in a HttpProxyOptions object. The file with modified code is attached. That code results in the following when a proper proxy configuration is given:

dev shell : 2.3.9 : default profile
$ poetry run python .../test-transcription.py
DEBUG: wrap_future
DEBUG: make_request
DEBUG: _get_connection
DEBUG: _create_connection
DEBUG: parsing proxy string: http://proxy.companyname.net:10443
DEBUG: parsed proxy_protocol: http
DEBUG: parsed proxy_user: None
DEBUG: parsed proxy_password: None
DEBUG: parsed proxy_host: proxy.companyname.net
DEBUG: parsed proxy_port: 10443
DEBUG: wrap_future
** Exception 0xc0000005 occured **
at 0x00007FFC6BF17E42: Failed to lookup symbol: error 487
at 0x00007FFC6BF1802F: Failed to lookup symbol: error 487
at 0x7FFC6BF2FB60: PyInit__awscrt
at 0x7FFC9B1F60A0: UnhandledExceptionFilter
at 0x7FFC9E05C500: memset
at 0x7FFC9E0466F0: _C_specific_handler
at 0x7FFC9E05A600: _chkstk
at 0x7FFC9DFE4550: RtlImageNtHeaderEx
at 0x7FFC9E059850: KiUserExceptionDispatcher
at 0x00007FFC6BED2BED: Failed to lookup symbol: error 487
at 0x00007FFC6BED2ACC: Failed to lookup symbol: error 487
at 0x00007FFC6BEBE7E7: Failed to lookup symbol: error 487
at 0x00007FFC6BEC64D0: Failed to lookup symbol: error 487
at 0x00007FFC6BEC78E8: Failed to lookup symbol: error 487
at 0x00007FFC6BEF18F3: Failed to lookup symbol: error 487
at 0x00007FFC6BEF03F5: Failed to lookup symbol: error 487
at 0x00007FFC6BEEE29A: Failed to lookup symbol: error 487
at 0x00007FFC6BECF2C5: Failed to lookup symbol: error 487
at 0x00007FFC6BECD371: Failed to lookup symbol: error 487
at 0x00007FFC6BECC84B: Failed to lookup symbol: error 487
at 0x00007FFC6BF0C53A: Failed to lookup symbol: error 487
at 0x00007FFC6BF0C745: Failed to lookup symbol: error 487
at 0x00007FFC6BF0B094: Failed to lookup symbol: error 487
at 0x00007FFC6BF05318: Failed to lookup symbol: error 487
at 0x00007FFC6BF21F62: Failed to lookup symbol: error 487
at 0x7FFC9C4784C0: BaseThreadInitThunk
at 0x7FFC9E001770: RtlUserThreadStart

httpsession.zip

[nateprewitt]

Hi @YichiRockyZhang,

Thanks for the detailed findings! So this looks like we have something strange going on in the CRT that we may not have control over in the SDK. If you don't include the proxy options, do you get the same failure at wrap_future with your test function?

If not, we may need to open a awscrt repository for a deeper inspection.

[YichiRockyZhang]

Thanks for the quick response @nateprewitt! When I omit proxy_options, I get a different error at wrap_future: awscrt.exceptions.AwsCrtError: AWS_IO_SOCKET_TIMEOUT: socket operation timed out. The trace is in my first code snippet above. I would guess the first error should be addressed to some extent in this code because there's no way to specify a proxy, and maybe the second error requires addressing in both here and awscrt.

[nateprewitt]

Yep, completely agree. Would you mind following up in an issue with the CRT team for the second issue? Once we have an idea of why that's dumping we can look at getting HttpProxyOptions integrated here.

[YichiRockyZhang]

I've submitted the issue on their side and looking to hear back. Thank you again for the support!

[TingDaoK]

Hi @juanmol, the AWS CRT has added provisional support for HTTP_PROXY and HTTPS_PROXY environment variables (awslabs/aws-crt-python#313) in awscrt==0.12.3.

If you get a moment to check this out, we'd be interested to know if this solves your issues. Thanks!

awscrt did add support for config proxy from environment variables from the native code. However, it's turned off by default. https://github.com/awslabs/aws-c-http/blob/main/include/aws/http/proxy.h#L39, as we don't want to break the default behavior before the support.

So, to properly support proxy configuration from environment variable support, it's gonna be a feature request for awscrt to expose the settings and used from this package.

[rktummalapenta]

Hi @TingDaoK , any update on the proxy issue, I am facing similar issue where its not picking up the proxy environment variables to test the aws transcribe streaming.