New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Disable Learning to Rank's involvement in queries #2905

Merged

sihugh merged 1 commit into main from ltr-removal

May 14, 2024

Contributor

sihugh commented May 9, 2024

Learning to Rank only changes the ordering of results when the query includes keywords and is ordered by relevance.

It can only affect how results rank within a single page. The maximum effect it could have is to reverse the order of a page of results. The results all still appear on the same page.

Site search searches with keywords (the majority of searches) are now served by Vertex AI search. The cost and complexity of running a Learning to Rank training and serving pipeline for the other use cases no longer seems commensurate with the impact.

In addition to this, the training of our model has not been supervised effectively for a while, leading to a drop in the quality of the model.

I therefore propose disabling the use of Learning to Rank in Search-API (this change) which allows the removal of the code to invoke the model, the pipeline that builds and deploys the model and the Terraform to support the deployment.


          Disable Learning to Rank's involvement in queries

67d42bb

Learning to Rank only changes the ordering of results when the query includes
keywords and is ordered by relevance.

It can only affect how results rank within a single page. The maximum effect it
could have is to reverse the order of a page of results. The results all still
appear on the same page.

Site search searches with keywords (the majority of searches) are now served by
Vertex AI search. The cost and complexity of running a Learning to Rank training
and serving pipeline for the other use cases no longer seems commensurate
with the impact.

In addition to this, the training of our model has not been supervised
effectively for a while, leading to a drop in the quality of the model.

I therefore propose disabling the use of Learning to Rank in Search-API
(this change) which allows the removal of the code to invoke the model, the
pipeline that builds and deploys the model and the Terraform to support the
deployment.

Contributor Author

sihugh commented May 9, 2024

A side effect of this is a fall in response times for keyword queries (in particular) because we:

won't have the extra network call to SageMaker (immediate)
can request less information from ElasticSearch on each query because we don't need to send it to the reranker (future?)

csutter approved these changes

View reviewed changes

Contributor

csutter left a comment

🙏

Contributor

KludgeKML commented May 9, 2024

This also starts clearing up the tech debt where marking the ab_test parameter is the only way to turn off LTR on an individual finder (the finder that replaced licence-finder had to do this to make the results usable)

KludgeKML approved these changes

View reviewed changes

leenagupte approved these changes

View reviewed changes

Contributor

leenagupte left a comment

🎉 🏅 🏆

sihugh merged commit f295032 into main

11 checks passed

sihugh deleted the ltr-removal branch

May 14, 2024 09:22

sihugh mentioned this pull request

Build learn-to-rank image on release. #2695

Closed

sihugh added a commit that referenced this pull request


          Remove LTR debug values

5726b69

These could be viewed in finder pages by appending `&debug_score=1` to the query string.

Finder-frontend attempts to access these defensively, so they are safe to remove.

[Learning to rank is being removed since this change](#2905)

sihugh added a commit that referenced this pull request


          Remove relevance_helpers

6e6b95c

[Learning to rank is being removed since this change](#2905)

sihugh added a commit that referenced this pull request


          Remove QueryParameters.rerank as it's always false

bbfb367

[Learning to rank is being removed since this change](#2905)

sihugh added a commit that referenced this pull request


          Remove reranker healthcheck

e7c3827

We don't need to monitor this, as it's not used any more.

[Learning to rank is being removed since this change](#2905)

sihugh added a commit that referenced this pull request


          Remove lib/learn_to_rank

df03877

[Learning to rank is being removed since this change](#2905)

sihugh added a commit that referenced this pull request


          Remove code reference to reranking

c929da1

[Learning to rank is being removed since this change](#2905)

sihugh added a commit that referenced this pull request


          Remove build/CI pipelines for LTR

a30148b

We don't need these any more. [Learning to rank is being removed since this change](#2905)

sihugh added a commit that referenced this pull request


          Remove LTR rake tasks

3c4ebfc

[Learning to rank is being removed since this change](#2905)

sihugh added a commit that referenced this pull request


          Remove Sagemaker gems

9219cde

These were only used as part of Learning to Rank.

[Learning to rank is being removed since this change](#2905)

This was referenced May 14, 2024

Start removing Learning to Rank bits alphagov/govuk-helm-charts#2037

Merged

Remove Learning to Rank code #2910

Merged

sihugh added a commit that referenced this pull request


          Remove LTR debug values

c5d6a40

These could be viewed in finder pages by appending `&debug_score=1` to the query string.

Finder-frontend attempts to access these defensively, so they are safe to remove.

[Learning to rank is being removed since this change](#2905)

sihugh added a commit that referenced this pull request


          Remove relevance_helpers

3420dcf

[Learning to rank is being removed since this change](#2905)

sihugh added a commit that referenced this pull request


          Remove QueryParameters.rerank as it's always false

c94a3a2

[Learning to rank is being removed since this change](#2905)

sihugh added a commit that referenced this pull request


          Remove reranker healthcheck

8c480de

We don't need to monitor this, as it's not used any more.

[Learning to rank is being removed since this change](#2905)

sihugh added a commit that referenced this pull request


          Remove lib/learn_to_rank

436c87e

[Learning to rank is being removed since this change](#2905)

sihugh added a commit that referenced this pull request


          Remove code reference to reranking

1eab60c

[Learning to rank is being removed since this change](#2905)

sihugh added a commit that referenced this pull request


          Remove build/CI pipelines for LTR

7b05332

We don't need these any more. [Learning to rank is being removed since this change](#2905)

sihugh added a commit that referenced this pull request


          Remove LTR rake tasks

20c3d4c

[Learning to rank is being removed since this change](#2905)

sihugh added a commit that referenced this pull request


          Remove Sagemaker gems

4228b7c

These were only used as part of Learning to Rank.

[Learning to rank is being removed since this change](#2905)

sihugh added a commit that referenced this pull request


          Remove LTR debug values

4de89c9

These could be viewed in finder pages by appending `&debug_score=1` to the query string.

Finder-frontend attempts to access these defensively, so they are safe to remove.

[Learning to rank is being removed since this change](#2905)

sihugh added a commit that referenced this pull request


          Remove relevance_helpers

84bfa85

[Learning to rank is being removed since this change](#2905)

sihugh added a commit that referenced this pull request


          Remove QueryParameters.rerank as it's always false

572cf81

[Learning to rank is being removed since this change](#2905)

sihugh added a commit that referenced this pull request


          Remove reranker healthcheck

8d4d2f6

We don't need to monitor this, as it's not used any more.

[Learning to rank is being removed since this change](#2905)

sihugh added a commit that referenced this pull request


          Remove lib/learn_to_rank

77974e6

[Learning to rank is being removed since this change](#2905)

sihugh added a commit that referenced this pull request


          Remove code reference to reranking

f037f11

[Learning to rank is being removed since this change](#2905)

sihugh added a commit that referenced this pull request


          Remove build/CI pipelines for LTR

42fef0c

We don't need these any more. [Learning to rank is being removed since this change](#2905)

sihugh added a commit that referenced this pull request


          Remove LTR rake tasks

df2ce9c

[Learning to rank is being removed since this change](#2905)

sihugh added a commit that referenced this pull request


          Remove Sagemaker gems

2abd359

These were only used as part of Learning to Rank.

[Learning to rank is being removed since this change](#2905)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment