Add tooling for recording and replaying PyPI interactions #609
Conversation
zanieb
force-pushed
the
zb/offlinepi
branch
2 times, most recently
from
c3520ff
to
9838342
Compare
| reqwest = { version = "0.11.22", default-features = false, features = ["json", "gzip", "brotli", "stream", "rustls-tls"] } | ||
| reqwest = { version = "0.11.22", default-features = false, features = ["json", "gzip", "brotli", "stream", "rustls-tls-native-roots"] } |
There was a problem hiding this comment.
I believe this is required because I registered the cert with my system.
There was a problem hiding this comment.
Suggested in seanmonstar/reqwest#1554
There was a problem hiding this comment.
rustls-tls: Enables TLS functionality provided by rustls. Equivalent to rustls-tls-webpki-roots.
rustls-tls-webpki-roots: Enables TLS functionality provided by rustls, while using root certificates from the webpki-roots crate.
rustls-tls-native-roots: Enables TLS functionality provided by rustls, while using root certificates from the rustls-native-certs crate.
source
Hm...
There was a problem hiding this comment.
|
Oh also I needed to install from GitHub to get things working initially, I should verify and document what version we require. |
Built on #609 When activating the `puffin-test-custom-ca-cert` feature, you can inject a custom ssl certificate by setting `PUFFIN_TEST_CA_CERT_PEM` to a pem file, e.g. ```bash PUFFIN_TEST_CA_CERT_PEM=$(pwd)/mitmproxy-ca-cert.pem ./offlinepi record cargo test --features pypi --features puffin-test-custom-ca-cert -- --test-threads=1 ``` This feature is off by default, so this is not possible in release builds.
|
A HAR file for our test suite is about 10 MB e.g. https://gist.github.com/zanieb/fb1d4f03176be020a2d0912476d19a1c |
Built on #609 When activating the `puffin-test-custom-ca-cert` feature, you can inject a custom ssl certificate by setting `PUFFIN_TEST_CA_CERT_PEM` to a pem file, e.g. ```bash PUFFIN_TEST_CA_CERT_PEM=$(pwd)/mitmproxy-ca-cert.pem ./offlinepi record cargo test --features pypi --features puffin-test-custom-ca-cert -- --test-threads=1 ``` This feature is off by default, so this is not possible in release builds.
|
I'll revert my cert changes in favor of #615 if possible and document usage. |
Closes #1474 Using the `rustls-tls-native-roots` feature > `rustls-tls`: Enables TLS functionality provided by rustls. Equivalent to rustls-tls-webpki-roots. > > `rustls-tls-webpki-roots`: Enables TLS functionality provided by rustls, while using root certificates from the webpki-roots crate. > > `rustls-tls-native-roots`: Enables TLS functionality provided by rustls, while using root certificates from the rustls-native-certs crate. Additional context: - seanmonstar/reqwest#1554 - encode/httpx#302 - [Should I use the native certs or webpki-roots?](https://github.com/rustls/rustls-native-certs#should-i-use-this-or-webpki-roots) Prior discussion at #609
Uses mitmproxy to record and replay interactions with a PyPI server. For example:
Then run tests with replayed responses:
I tested this with my internet turned off, so it looks to be fully offline.
See the README for more details.
I want to figure out a good way to help people install the certificate as it's not trivial. I did:
See #615 for a more general solution.
I'm also not sure what the best way is to hook this into our test suite in practice. It'd be nice to have a command that recorded new responses automatically or just replayed if not needed, but it's unclear how that'd be done :) it seems best for others to play with this before we make it integral in any way.