Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Replace all test private keys with dynamically-generated keys like //.hierarchy #7476

Open
21 of 24 tasks
aarongable opened this issue May 7, 2024 · 3 comments
Open
21 of 24 tasks
Assignees

Comments

@aarongable
Copy link
Contributor

aarongable commented May 7, 2024

Twice in the last two weeks we have uploaded new private keys in PRs, and had to have SRE block those keys because they're compromised.

This should be impossible. We should make it so that boulder has no need for checked-in keys, by dynamically generating any such keys when the integration tests start. Then we should enable push protection with secret scanning to ensure that we never upload a private key to this repo ever again.

$ ag -l "BEGIN.*PRIVATE KEY"
  • test/test-root2.key
  • test/test-root.key
  • test/hierarchy/*.key.pem
  • test/grpc-creds/*/key.pem
  • test/test-ee.key
  • test/test-example.key
  • test/mail-test-srv/localhost/key.pem
  • test/mail-test-srv/minica-key.pem
  • test/redis-tls/minica-key.pem
  • test/redis-tls/boulder/key.pem
  • test/redis-tls/boulder-redis/key.pem
  • test/redis-tls/redis/key.pem
  • test/wfe-tls/minica-key.pem
  • test/wfe-tls/boulder/key.pem
  • ca/testdata/ca_key.pem
  • ca/testdata/testcsr.go
  • cmd/ocsp-responder/testdata/test-ca.key
  • grpc/creds/testdata/example.com/key.pem
  • wfe2/test/238.key
  • wfe2/test/178.key
  • wfe2/wfe_test.go
$ find -type f -iname '*.der'
  • //test/test-root.key.der
  • //test/test-key-5.der
  • //test/test-ca.key.der
@pgporada
Copy link
Member

pgporada commented May 7, 2024

There's also some DER encoded private keys floating around such as the following. Note that public keys haven't been sorted out from this.

$ find -type f -iname '*.der'
./test/test-root.key.der
./test/test-ca.der
./test/test-key-5.der
./test/test-ca.key.der
./test/test-root.der

@aarongable
Copy link
Contributor Author

test-ca.der and test-root.der are certificates; I've added the others to the checklist above.

@aarongable aarongable self-assigned this May 14, 2024
@aarongable aarongable added this to the Sprint 2024-05-14 milestone May 14, 2024
pgporada pushed a commit that referenced this issue May 15, 2024
Delete a variety of unused or lightly-used private keys, along with any
dangling references to them.

Part of #7476
beautifulentropy pushed a commit that referenced this issue May 15, 2024
The summary here is:
- Move test/cert-ceremonies to test/certs
- Move .hierarchy (generated by the above) to test/certs/webpki
- Remove our mapping of .hierarchy to /hierarchy inside docker
- Move test/grpc-creds to test/certs/ipki
- Unify the generation of both test/certs/webpki and test/certs/ipki
into a single script at test/certs/generate.sh
- Make that script the entrypoint of a new docker compose service
- Have t.sh and tn.sh invoke that service to ensure keys and certs are
created before tests run

No production changes are necessary, the config changes here are just
for testing purposes.

Part of #7476
@aarongable
Copy link
Contributor Author

aarongable commented May 15, 2024

With #7488 out for review, only three clusters of keys remains:

  • test-key-5 and the keys hardcoded into wfe_test.go, which correspond to pubkeys hardcoded into mocks/mocks.go and will therefore be a bit more complicated to remove
  • the minica keys and certs for mail-test-srv, redis-tls, and wfe-tls, which can largely just be moved into the dynamically-generated PKI, except that some of them are used by unittests too
  • //test/hierarchy, which we use nearly everywhere, and have been discussing replacing with a test-only package that generates such certs in-memory at init() time.

pgporada pushed a commit that referenced this issue May 16, 2024
aarongable added a commit that referenced this issue May 17, 2024
Remove the redis-tls, wfe-tls, and mail-test-srv keys which were
generated by minica and then checked in to the repo. All three are
replaced by the dynamically-generated ipki directory.

Part of #7476
aarongable added a commit that referenced this issue May 21, 2024
Replace "mocks.StorageAuthority" with "sapb.StorageAuthorityClient" in
our test mocks. The improves them by removing implementations of the
methods the tests don't actually need, instead of inheriting lots of
extraneous methods from the huge and cumbersome mocks.StorageAuthority.

This reduces our usage of mocks.StorageAuthority to only the WFE tests
(which create one in the frequently-used setup() function), which will
make refactoring those mocks in the pursuit of
#7476 much easier.

Part of #7476
aarongable added a commit that referenced this issue May 24, 2024
Update the version of protoc-gen-go-grpc that we use to generate Go gRPC
code from our proto files, and update the versions of other gRPC tools
and libraries that we use to match. Turn on the new
`use_generic_streams` code generation flag to change how
protoc-gen-go-grpc generates implementations of our streaming methods,
from creating a wholly independent implementation for every stream to
using shared generic implementations.

Take advantage of this code-sharing to remove our SA "wrapper" methods,
now that they have truly the same signature as the SARO methods which
they wrap. Also remove all references to the old-style stream names
(e.g. foopb.FooService_BarMethodClient) and replace them with the new
underlying generic names, for the sake of consistency. Finally, also
remove a few custom stream test mocks, replacing them with the generic
mocks.ServerStreamClient.

Note that this PR does not change the names in //mocks/sa.go, to avoid
conflicts with work happening in the pursuit of
#7476. Note also that this
PR updates the version of protoc-gen-go-grpc that we use to a specific
commit. This is because, although a new release of grpc-go itself has
been cut, the codegen binary is a separate Go module with its own
releases, and it hasn't had a new release cut yet. Tracking for that is
in grpc/grpc-go#7030.
vbaranovskiy-plesk pushed a commit to plesk/boulder that referenced this issue May 30, 2024
…ypt#7477)

The summary here is:
- Move test/cert-ceremonies to test/certs
- Move .hierarchy (generated by the above) to test/certs/webpki
- Remove our mapping of .hierarchy to /hierarchy inside docker
- Move test/grpc-creds to test/certs/ipki
- Unify the generation of both test/certs/webpki and test/certs/ipki
into a single script at test/certs/generate.sh
- Make that script the entrypoint of a new docker compose service
- Have t.sh and tn.sh invoke that service to ensure keys and certs are
created before tests run

No production changes are necessary, the config changes here are just
for testing purposes.

Part of letsencrypt#7476
vbaranovskiy-plesk pushed a commit to plesk/boulder that referenced this issue May 30, 2024
vbaranovskiy-plesk pushed a commit to plesk/boulder that referenced this issue May 30, 2024
Remove the redis-tls, wfe-tls, and mail-test-srv keys which were
generated by minica and then checked in to the repo. All three are
replaced by the dynamically-generated ipki directory.

Part of letsencrypt#7476
vbaranovskiy-plesk pushed a commit to plesk/boulder that referenced this issue May 30, 2024
Replace "mocks.StorageAuthority" with "sapb.StorageAuthorityClient" in
our test mocks. The improves them by removing implementations of the
methods the tests don't actually need, instead of inheriting lots of
extraneous methods from the huge and cumbersome mocks.StorageAuthority.

This reduces our usage of mocks.StorageAuthority to only the WFE tests
(which create one in the frequently-used setup() function), which will
make refactoring those mocks in the pursuit of
letsencrypt#7476 much easier.

Part of letsencrypt#7476
vbaranovskiy-plesk pushed a commit to plesk/boulder that referenced this issue May 30, 2024
Update the version of protoc-gen-go-grpc that we use to generate Go gRPC
code from our proto files, and update the versions of other gRPC tools
and libraries that we use to match. Turn on the new
`use_generic_streams` code generation flag to change how
protoc-gen-go-grpc generates implementations of our streaming methods,
from creating a wholly independent implementation for every stream to
using shared generic implementations.

Take advantage of this code-sharing to remove our SA "wrapper" methods,
now that they have truly the same signature as the SARO methods which
they wrap. Also remove all references to the old-style stream names
(e.g. foopb.FooService_BarMethodClient) and replace them with the new
underlying generic names, for the sake of consistency. Finally, also
remove a few custom stream test mocks, replacing them with the generic
mocks.ServerStreamClient.

Note that this PR does not change the names in //mocks/sa.go, to avoid
conflicts with work happening in the pursuit of
letsencrypt#7476. Note also that this
PR updates the version of protoc-gen-go-grpc that we use to a specific
commit. This is because, although a new release of grpc-go itself has
been cut, the codegen binary is a separate Go module with its own
releases, and it hasn't had a new release cut yet. Tracking for that is
in grpc/grpc-go#7030.
vbaranovskiy-plesk pushed a commit to plesk/boulder that referenced this issue May 30, 2024
…ypt#7477)

The summary here is:
- Move test/cert-ceremonies to test/certs
- Move .hierarchy (generated by the above) to test/certs/webpki
- Remove our mapping of .hierarchy to /hierarchy inside docker
- Move test/grpc-creds to test/certs/ipki
- Unify the generation of both test/certs/webpki and test/certs/ipki
into a single script at test/certs/generate.sh
- Make that script the entrypoint of a new docker compose service
- Have t.sh and tn.sh invoke that service to ensure keys and certs are
created before tests run

No production changes are necessary, the config changes here are just
for testing purposes.

Part of letsencrypt#7476
vbaranovskiy-plesk pushed a commit to plesk/boulder that referenced this issue May 31, 2024
…ypt#7477)

The summary here is:
- Move test/cert-ceremonies to test/certs
- Move .hierarchy (generated by the above) to test/certs/webpki
- Remove our mapping of .hierarchy to /hierarchy inside docker
- Move test/grpc-creds to test/certs/ipki
- Unify the generation of both test/certs/webpki and test/certs/ipki
into a single script at test/certs/generate.sh
- Make that script the entrypoint of a new docker compose service
- Have t.sh and tn.sh invoke that service to ensure keys and certs are
created before tests run

No production changes are necessary, the config changes here are just
for testing purposes.

Part of letsencrypt#7476
vbaranovskiy-plesk pushed a commit to plesk/boulder that referenced this issue May 31, 2024
…ypt#7477)

The summary here is:
- Move test/cert-ceremonies to test/certs
- Move .hierarchy (generated by the above) to test/certs/webpki
- Remove our mapping of .hierarchy to /hierarchy inside docker
- Move test/grpc-creds to test/certs/ipki
- Unify the generation of both test/certs/webpki and test/certs/ipki
into a single script at test/certs/generate.sh
- Make that script the entrypoint of a new docker compose service
- Have t.sh and tn.sh invoke that service to ensure keys and certs are
created before tests run

No production changes are necessary, the config changes here are just
for testing purposes.

Part of letsencrypt#7476
vbaranovskiy-plesk pushed a commit to plesk/boulder that referenced this issue May 31, 2024
vbaranovskiy-plesk pushed a commit to plesk/boulder that referenced this issue May 31, 2024
Remove the redis-tls, wfe-tls, and mail-test-srv keys which were
generated by minica and then checked in to the repo. All three are
replaced by the dynamically-generated ipki directory.

Part of letsencrypt#7476
vbaranovskiy-plesk pushed a commit to plesk/boulder that referenced this issue May 31, 2024
Replace "mocks.StorageAuthority" with "sapb.StorageAuthorityClient" in
our test mocks. The improves them by removing implementations of the
methods the tests don't actually need, instead of inheriting lots of
extraneous methods from the huge and cumbersome mocks.StorageAuthority.

This reduces our usage of mocks.StorageAuthority to only the WFE tests
(which create one in the frequently-used setup() function), which will
make refactoring those mocks in the pursuit of
letsencrypt#7476 much easier.

Part of letsencrypt#7476
vbaranovskiy-plesk pushed a commit to plesk/boulder that referenced this issue May 31, 2024
Update the version of protoc-gen-go-grpc that we use to generate Go gRPC
code from our proto files, and update the versions of other gRPC tools
and libraries that we use to match. Turn on the new
`use_generic_streams` code generation flag to change how
protoc-gen-go-grpc generates implementations of our streaming methods,
from creating a wholly independent implementation for every stream to
using shared generic implementations.

Take advantage of this code-sharing to remove our SA "wrapper" methods,
now that they have truly the same signature as the SARO methods which
they wrap. Also remove all references to the old-style stream names
(e.g. foopb.FooService_BarMethodClient) and replace them with the new
underlying generic names, for the sake of consistency. Finally, also
remove a few custom stream test mocks, replacing them with the generic
mocks.ServerStreamClient.

Note that this PR does not change the names in //mocks/sa.go, to avoid
conflicts with work happening in the pursuit of
letsencrypt#7476. Note also that this
PR updates the version of protoc-gen-go-grpc that we use to a specific
commit. This is because, although a new release of grpc-go itself has
been cut, the codegen binary is a separate Go module with its own
releases, and it hasn't had a new release cut yet. Tracking for that is
in grpc/grpc-go#7030.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants