Unintended protocol prioritisation with scheme list

[dynamic-entropy]

Description

On upgrading to Rucio 34, we observed gridFTP was being used with priority, instead of the domain priority configuration of protocols in the RSE, which is davs for all of our sites.

Protocol priority for our sites

davs: 1
gsiftp: 2
root: 3

The conveyor configuration at the time

[conveyor]
scheme = srm,gsiftp,root,http,https

We suspect either of the two scenarios:

1. Fix to use the configured scheme list (submitter doesn't use the configured scheme list #6092) filtered out davs
   • Note: davs is missing from the above list but https is specified
   • However: we had transfers active over davs at the same time, as opposed to having protocol_mistmatch errors
2. The list is being used to sort the transfers somehow
   • Note: my inspection of the code did not find anything of that sort
     

     rucio/lib/rucio/rse/rsemanager.py

     Lines 702 to 708 in 695c4bc

     	dest_candidates = sorted(dest_candidates, key=lambda k: k['domains'][domain][operation_dest])
     	src_candidates = sorted(src_candidates, key=lambda k: k['domains'][domain][operation_src])
     	for dest_protocol in dest_candidates:
     	for src_protocol in src_candidates:
     	if __check_compatible_scheme(dest_protocol['scheme'], src_protocol['scheme']):
     	return (dest_protocol['scheme'], src_protocol['scheme'], dest_protocol['domains'][domain][operation_dest], src_protocol['domains'][domain][operation_src])

For now, we have changed the scheme list to get out desired priority

[conveyor]
scheme = davs, root, gsiftp, srm

Steps to reproduce

While on Rucio 34

1. Use the below configuration for both src_rse and dest_rse.

Protocols:
==========
  davs
    domains: '{"lan": {"read": 0, "write": 0, "delete": 0}, "wan": {"read": 1, "write": 1, "delete": 1, "third_party_copy_read": 1, "third_party_copy_write": 1}}'
    extended_attributes: None
    hostname: <src|dst hostname>
    impl: rucio.rse.protocols.gfal.Default
    port: <src|dst port >
    prefix: /
    scheme: davs
  gsiftp
    domains: '{"lan": {"read": 0, "write": 0, "delete": 0}, "wan": {"read": 2, "write": 2, "delete": 2, "third_party_copy_read": 2, "third_party_copy_write": 2}}'
    extended_attributes: None
    hostname: <src|dst hostname>
    impl: rucio.rse.protocols.gfal.Default
    port: <src|dst port >
    prefix: /
    scheme: gsiftp

2. Set the following scheme list

[conveyor]
scheme = gsiftp, davs

3. Observe the chosen scheme for transfers

Rucio Version

Server: 34.0.0

Additional Information

No response