feat: Resolving CORS Restrictions in Node.js

[yixinNB]

🚀 Feature

Resolve CORS Restrictions in Node.js

Motivation

In Node.js, I'm attempting to employ Pyodide for executing Python scripts uploaded by users, leveraging its strong isolation environment. However, certain Python scripts might employ the requests library to scrape web pages. While WebAssembly theoretically doesn't face CORS limitations in Node.js, Pyodide continues to encounter errors during runtime.

Pitch

Crawling web pages with python in nodejs and pyodide environments without cors limitations

Alternatives

I considered CORS proxy, but I would like to run user uploaded python code in pyodide as if it were running locally, without special handling. Additionally, I would like to increase the security of the code by using pyodide's sandboxing mechanism and nodejs' sandboxing mechanism (e.g. vm2).

Additional context

/lib/python3.11/site-packages/urllib3/connectionpool.py:1101: InsecureRequestWarning: Unverified HTTPS request is being made to host 'api.live.bilibili.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#tls-warnings
  warnings.warn(
Can't stream HTTP requests because: 
  Page is not cross-origin isolated
 Worker or Blob classes are not available in this environment.

[hoodmane]

Thanks for the report @yixinNB! think this should be urllib3 issue. Cc @joemarshall.

[joemarshall]

This is a urllib3 warning.

Requests will in theory work fine as long as you have a package that implements xmlhttprequest loaded, but you won't be able to stream data, these http requests will be downloaded then the full body returned to python in one go.

It would be possible to implement support for node.js streaming requests in urllib3 (and thus requests) on versions of node with worker-threads and atomics support, which I think pretty much anything that reasonably supports pyodide would run happily. If you could find funding for it I could do it (I wrote the emscripten browser support for urllib3), I don't think it would be masses of work.