design: tech debt: e2e tests refactoring

[tzaffi]

Problem

The end to end tests are an important sanity check ensuring that the latest version of indexer is production-ready.

However, they suffer from a few issues that make them difficult to work with and identify/fix problems when they occur:

• weave together disparate components:
  • AWS tarballs to define a network to test against
  • docker containers for test environmenting
  • python library to execute processes and validate results

These components have become bloated:

• docker:
  • over-duplication across multiple docker-compose.yml's and Dockerfile's
• python:
  • old and bespoke tests that are very difficult to modify and modernize
  • favor speed (via multi-processing) over clarity and ease of use
  • xrun may not even work correctly. @bbroder-algo pointed out in a discussion in Enhancement: more logging in E2E tests + turn back on commit-workflow E2E test #1523 that the time out may not engage: https://stackoverflow.com/questions/36952245/subprocess-timeout-failure

As a result:

• adding new tests to precisely validate new features becomes tedious and basically isn't done
• diagnosing and fixing a break can take hours/days instead of minutes for the uninitiated

Solution

Produce a design doc that proposes a way to refactor the end to end tests so as to fix these issues.

Dependencies

None

Urgency

Medium

[winder]

Maybe these old tests can be deprecated in favor of the new e2e scripts used for conduit: https://github.com/algorand/conduit/tree/master/e2e_tests

[tzaffi]

Maybe these old tests can be deprecated in favor of the new e2e scripts used for conduit: https://github.com/algorand/conduit/tree/master/e2e_tests

They seem to share some python functions, but I agree that we should look to see if we can consolidate the python if possible.