Remove unused index idx_last_scheduling_decision on dag_run table

[pankajkoti]

We added idx_last_scheduling_decision on the
last_scheduling_decision column in the dag_run table
with Airflow 2.0.0. However, this index seems to have been
unused with 0 index_scan counts throughout. I verified
scheduler performance metrics are not affected after removing
this index.

Additionally, this index seems to occupy huge storage space,
almost half the size of the table (e.g. on one deployment that
I checked, the table occupies 9.8GB of records, but this single
index alone occupies 5.7 GB)

---

^ Add meaningful description above
Read the Pull Request Guidelines for more information.
In case of fundamental code changes, an Airflow Improvement Proposal (AIP) is needed.
In case of a new dependency, check compliance with the ASF 3rd Party License Policy.
In case of backwards incompatible changes please leave a note in a newsfragment file, named {pr_number}.significant.rst or {issue_number}.significant.rst, in newsfragments.

[Taragolis]

single index alone occupies 5.7 GB

That a bit strange that this index could use such amount of storage. Maybe this happen due to disabled auto vacuum daemon or never.

In general I agree that this index unlikely will ever use in Postgres. There is couple places where last_scheduling_decision use in sorting or filtration, but every time it use alongside with other fields so this much less effective rather than even make seq scan or use other indexes for optimise particular queries.

So removal of this one could make more benefits rather than keep it

[pankajkoti]

@vincbeck @Taragolis @jedcunningham @utkarsharma2 addressed the review comments so far. Requesting re-review please.

[vincbeck]

Much better! Thanks!

[pankajkoti]

Bringing up a hidden in a resolved conversation comment from Jed in a conversation so that it appears here upfront

From Jed
"cc @ephraimbuddy we will want to remove 2.10.0 when we cherry-pick this back to v2-9."

The comment talks about removing this line https://github.com/apache/airflow/pull/39275/files#diff-730eff695b18ad05249293cc3361da7424aba164857e9c5d26e50bd3a78ba6e7R95 when cherry-picking back to v2-9

[ephraimbuddy]

Bringing up a hidden in a resolved conversation comment from Jed in a conversation so that it appears here upfront

From Jed "cc @ephraimbuddy we will want to remove 2.10.0 when we cherry-pick this back to v2-9."

The comment talks about removing this line https://github.com/apache/airflow/pull/39275/files#diff-730eff695b18ad05249293cc3361da7424aba164857e9c5d26e50bd3a78ba6e7R95 when cherry-picking back to v2-9

Pre-commit will still fix it when we cherry pick. Nothing to worry about