Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tests: generate the indexes in the HTML search JavaScript tests using the application code. #12099

Open
jayaddison opened this issue Mar 15, 2024 · 2 comments · May be fixed by #12102
Open

Tests: generate the indexes in the HTML search JavaScript tests using the application code. #12099

jayaddison opened this issue Mar 15, 2024 · 2 comments · May be fixed by #12102
Assignees
Labels
html search internals:refactoring javascript Pull requests that update Javascript code type:proposal a feature suggestion type:tests

Comments

@jayaddison
Copy link
Contributor

jayaddison commented Mar 15, 2024

Is your feature request related to a problem? Please describe.
We have JavaScript test coverage of our HTML search functionality, and that's great.

However: the tests use inline-declared index datastructures to represent what each test index is expected to contain.

That means that the format of the search indexes produced by the application code could go out of sync with the JavaScript tests unnoticed, and we may not catch regressions -- or, equally as annoyingly, it might be very difficult to accurately write additional tests from scratch in future.

Describe the solution you'd like
Although it would add Python, and sphinx, and its requirements to the dependencies for the JavaScript test workflow, I think we should do that and build test index fixtures dynamically.

They should be checked-in to source control, and the test would fail if the generated-index didn't match the one currently checked-in.

An approach that does not require a Python runtime dependency for the JavaScript tests was found instead; the indices can be built using Python, then committed to source control and read as fixture data when the JavaScript tests run.

To catch changes to the index format, a unit test in the Python test suite attempts to regenerate them using the latest application code, and fails when differences are detected in the content.

Describe alternatives you've considered
Allowing the status-quo to continue seems fine, but the fact that we lack continuous committers / experts in the search area (it has taken me a while to begin to understand it, and I still don't feel completely confident about it) makes me want to guard it a bit against lack of continuous maintenance/knowledge.

Additional context
This would require good documentation too.

@jayaddison
Copy link
Contributor Author

jayaddison commented Mar 15, 2024

Design (as implemented in #12102):

  • Refactor the test codebase to extract test fixtures.
  • Serve the test fixtures using karma's built-in local webserver so that the indexes can be retrieved and loaded similarly to the way they would be during a Sphinx webbrowser session.
  • Reverse-engineer or create a few similar test projects to fill-in for the existing JavaScript search test cases. Place these roots under tests/js/roots so that they do not intermingle with commits to the Python test suite.
  • Add a script under the utils directory that regenerates the fixtures from source.
  • Add a GitHub Actions workflow that regenerates the test fixtures and fails if uncommitted git changes are detected as a result. Fail the Python test suite when freshly-regenerated search indexes do not match their corresponding source-controlled versions.
  • Add documentation to the codebase about how this works.

@jayaddison

This comment was marked as outdated.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
html search internals:refactoring javascript Pull requests that update Javascript code type:proposal a feature suggestion type:tests
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant