Fortify pre-release testing regime

[gwright99]

Background

Official Seqera deployment guidance for Tower implementations is that clients should use managed database & redis instances to ensure better data protection and application stability (i.e. RDS and Elasticache). This is a good idea.

The use of managed instances is problematic in testing scenarios, however, as it introduces significant delays into the testing cycle (~10+ minutes for stand-up, and often longer for teardown due in-place modifications to remove teardown protections on the database). Furthermore, use of the external instances incurs real-world charges from AWS for each testing cycle using the managed instances.

Given these realities, current internal testing tends to favour deployments which use the containerized db & redis. While this is often not a problem (e.g. the database implementation doesn't matter if you are testing conditional logic re: how the tower.yml is populated), it means the RDS/Elasticache logic is being exercised less on a daily basis and this can result in bugs being unintentionally introduced and not caught until later.

The simplest way to ensure more robustness of the project is to introduce a CICD solution which can run automated testing/checks against PRs / master branch. This would remove the cognitive burden from project resources to manually try alternative solutions and allow us to publish test results (which would allow us to support clients installing in more regulated / qualified environments).

Unfortunately, the introduction of CICD doesn't solve the underlying problems of:

1. Where are you going to test?
2. How much is it going to cost?
3. How long is it going to take?

Proposal

No matter what, we always need to run final sanity checks in a real AWS account to ensure things truly work the way we expect them in a true cloud environment. But we don't need to be exclusively testing in AWS 100%.

We should consider making better use of a localized testing harness to minimize costs and speed up testing execution. In the case of AWS, the use of Localstack (Pro) can provide much of the functionality we need on a daily basis (Terrafrom support, AWS API compliance, spawning of containers emulating the docker-compose host and associated Tower credentials).

I think it would be worthwhile to get an end-to-end process fully working, without requiring too much deviation from the official project (localstack pro still has some limitations which does not allow a 100% apples-to-apples parity).

Counter Arguments

A few arguments can be made against this effort. I have thoughts on each but won't comment yet as I'd like to see what others say:

1. This is supposed to be a field tool, why are we adding extra layers of complexity into a "use at your own discretion" tool?

2. Manual testing is fine - the solution is not big enough to warrant a more complex solution.

3. (Assuming use of localstack) the localstack solution just isn't comparable enough to real AWS to be worth investing in.

Next Steps

1. Discuss this topic with CX resources & Seqera leadership to determine our desired approach.

2. Implement trial effort to assess feasibility.

[gwright99]

The opening of this ticket was a bit of an after-the-fact thing, as I've already been working on getting a localstack implementation running and almost have the whole thing working on a local VM (e2e deployment ~1 minute instead of the ~15min going against real AWS).

There are still a few final problems to solve (e.g. localstack uses docker-in-docker, so the "ec2"container - while able to spawn the various Tower containers - sees the Tower containers failing due to not being able to find /tower.yml; some problems with Ansible package installation which required baking of packages into a custom image treated as the AMI).

Assuming these last few problems can be solved, we will have a working model that a simple CICD can be pointed against.