Preserve user supplied joins order as much as possible

[kamipo]

Currently, string joins are always applied as last joins part, and Arel
join nodes are always applied as leading joins part (since #36304), it
makes people struggled to preserve user supplied joins order.

To mitigate this problem, preserve the order of string joins and Arel
join nodes either before or after of association joins.

Fixes #36761.
Fixes #34328.
Fixes #24281.
Fixes #12953.

[eileencodes]

We're hitting an issue where this changed behavior in GitHub for Rails 6.0. I haven't quite tracked down what's happening except that there are extra joins in the query. We use a lot of chained scopes and often those scopes will join on tables that perhaps another scope joined on. Previously I believe this was just ignored. It's not as simple as chained scopes though and I haven't been able to reproduce outside of the app yet. I think there is a bug in this change.

One of our scopes used to create sql like this:

LEFT OUTER JOIN `table_a` `column_a`
ON `column_a`.`target_type` = 'User'
AND `column_a`.`target_id` = `column_b`.`id`
AND `column_a`.`name` = 'disable'
AND `column_a`.`value` = 'TRUE' INNER JOIN `table_b`
ON `table_b`.`issue_column_d_id` = `table_c`.`id` INNER JOIN `table_d`
ON `table_b`.`subject_type` = 'Project'
AND `table_b`.`subject_id` = `table_d`.`id` WHERE `table_c`.`column_c` = 53 AND `table_c`.`column_d` IN ('added') AND ((`table_d`.`owner_type` = 'User')

and now creates sql like this:

LEFT OUTER JOIN `table_a` `column_a`
  ON `column_a`.`target_type` = 'User'
 AND `column_a`.`target_id` = `column_b`.`id`
 AND `column_a`.`name` = 'disable'
 AND `column_a`.`value` = 'TRUE' WHERE `table_c`.`column_c` = 58 AND `table_c`.`column_d` IN ('added')
 AND ((`table_d`.`owner_type` = 'User')

There are 2 extra inner joins in the first one. Do you know what's going on here @kamipo?

This results in a not unique alias method.

[kamipo]

Weird... could you share the joins_values, left_outer_joins_values (just in case eager_load_values and includes_values) in the relation?

I guess the relation from the first one sql, it seems constructed by TableX(or TableB).left_joins(:table_a).merge(TableC.joins(table_b: :table_d)) (since Active Record has almost no way to apply LEFT OUTER JOIN before INNER JOIN).

If so, the values will be joins_values = [#<JoinDependency: for TableC.joins(table_b: :table_d)>], left_outer_joins_values = [:table_a].

If the INNER JOINs are lost, it seems that #<JoinDependency: for TableC.joins(table_b: :table_d)> is lost in joins_values or buckets[:stashed_join] somehow.

[eileencodes]

@kamipo omg I'm so sorry, I swapped the queries. The inner joins aren't lost, we have extras.
The query we have is gigantic and it's really hard to track down.

What I provided is one part of a bunch of queries that get unioned together. Earlier in the query we join table_b and table_d already, so the second attempt to join causes a unique table alias error.

I think (and haven't yet reproduced because this query is so complex) that Rails used to drop the extra joins and now it's keeping them around.

[codeodor]

@eileencodes did you all find a workaround for this issue in Rails 6? I'm running up against something similar: in one app I maintain, it just obliterates the left join produced by includes + references and in another app it moves it to below the additional joins we added, so that it is now out of order to where the columns don't exist in the query before being referenced.

[codeodor]

@eileencodes Never mind, I see #36834 seems to have fixed it for you. I guess I should probably open a new issue, since I think that made its way into Rails 6, and I'm still having a similar problem.