Files from root://submit50.mit.edu are not available at coffea-casa through XCache

[oshadura]

Files from root://submit50.mit.edu are not available at coffea-casa through XCache:

>>> import uproot
>>> uproot.open("root://xcache//store/user/mariadlf/nano/D02/GluGlu_HToPhiGamma_M125_TuneCP5_PSWeights_13TeV_powheg_pythia8+RunIISummer20UL18MiniAODv2-106X_upgrade2018_realistic_v16_L1v1-v3+MINIAODSIM/8EDDE712-E106-7B48-AB0B-7E4580426E8E.root")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/conda/lib/python3.9/site-packages/uproot/reading.py", line 141, in open
    file = ReadOnlyFile(
  File "/opt/conda/lib/python3.9/site-packages/uproot/reading.py", line 580, in __init__
    self._source = Source(
  File "/opt/conda/lib/python3.9/site-packages/uproot/source/xrootd.py", line 272, in __init__
    self._open()
  File "/opt/conda/lib/python3.9/site-packages/uproot/source/xrootd.py", line 275, in _open
    self._resource = XRootDResource(self._file_path, self._timeout)
  File "/opt/conda/lib/python3.9/site-packages/uproot/source/xrootd.py", line 83, in __init__
    self._open()
  File "/opt/conda/lib/python3.9/site-packages/uproot/source/xrootd.py", line 92, in _open
    self._xrd_error(status)
  File "/opt/conda/lib/python3.9/site-packages/uproot/source/xrootd.py", line 118, in _xrd_error
    raise OSError(
OSError: XRootD error: [ERROR] Operation expired
in file root://xcache//store/user/mariadlf/nano/D02/GluGlu_HToPhiGamma_M125_TuneCP5_PSWeights_13TeV_powheg_pythia8+RunIISummer20UL18MiniAODv2-106X_upgrade2018_realistic_v16_L1v1-v3+MINIAODSIM/8EDDE712-E106-7B48-AB0B-7E4580426E8E.root

I checked and I could open any other file:

>>> uproot.open("root://xcache//store/mc/RunIISummer20UL17NanoAODv2/TTToSemiLeptonic_TuneCP5_13TeV-powheg-pythia8/NANOAODSIM/106X_mc2017_realistic_v8-v1/120000/2A2F4EC9-F9BB-DF43-B08D-525B5389937E.root")
<ReadOnlyDirectory '/' at 0x7f171a2c23a0>

[oshadura]

@jthiltges do you have an idea what could be wrong?

[jthiltges]

I'm getting the impression that submit50.mit.edu storage isn't part of the CMS XRootD data federation. Looking at the MIT redirector, I'm not getting a result:

$ xrdfs xrootd.cmsaf.mit.edu ls /store/user/mariadlf
[ERROR] Server responded with an error: [3011] Too many attempts to gain dfs read access to the file

That /store/mc data file is available at Purdue and FNAL:

$ xrdfs xrootd.unl.edu locate '/store/mc/RunIISummer20UL17NanoAODv2/TTToSemiLeptonic_TuneCP5_13TeV-powheg-pythia8/NANOAODSIM/106X_mc2017_realistic_v8-v1/120000/2A2F4EC9-F9BB-DF43-B08D-525B5389937E.root'
af-a00.cms.rcac.purdue.edu:1098 Server ReadWrite 
cmsxrootd-site1.fnal.gov:1094 Manager Read
cmsxrootd-site3.fnal.gov:1094 Manager Read
cmssrv608.fnal.gov:1094 Manager Read
cmsxrootd-site2.fnal.gov:1094 Manager Read

It should work OK to access the submit50 data directly with the full URL to the storage server, but not through the xcache.

[oshadura]

@jthiltges it doesn't work also with full URL since submit50 doesn't support tokens:

cms-jovyan@jupyter-oksana-2eshadura-40cern-2ech:~$ python
Python 3.9.18 | packaged by conda-forge | (main, Aug 30 2023, 03:49:32) 
[GCC 12.3.0] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import uproot
>>> uproot.open("root://submit50.mit.edu//store/user/mariadlf/nano/D02/GluGlu_HToPhiGamma_M125_TuneCP5_PSWeights_13TeV_powheg_pythia8+RunIISummer20UL18MiniAODv2-106X_upgrade2018_realistic_v16_L1v1-v3+MINIAODSIM/8EDDE712-E106-7B48-AB0B-7E4580426E8E.root")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/opt/conda/lib/python3.9/site-packages/uproot/reading.py", line 141, in open
    file = ReadOnlyFile(
  File "/opt/conda/lib/python3.9/site-packages/uproot/reading.py", line 580, in __init__
    self._source = Source(
  File "/opt/conda/lib/python3.9/site-packages/uproot/source/xrootd.py", line 272, in __init__
    self._open()
  File "/opt/conda/lib/python3.9/site-packages/uproot/source/xrootd.py", line 275, in _open
    self._resource = XRootDResource(self._file_path, self._timeout)
  File "/opt/conda/lib/python3.9/site-packages/uproot/source/xrootd.py", line 83, in __init__
    self._open()
  File "/opt/conda/lib/python3.9/site-packages/uproot/source/xrootd.py", line 92, in _open
    self._xrd_error(status)
  File "/opt/conda/lib/python3.9/site-packages/uproot/source/xrootd.py", line 118, in _xrd_error
    raise OSError(
OSError: XRootD error: [FATAL] Auth failed: No protocols left to try
in file root://submit50.mit.edu//store/user/mariadlf/nano/D02/GluGlu_HToPhiGamma_M125_TuneCP5_PSWeights_13TeV_powheg_pythia8+RunIISummer20UL18MiniAODv2-106X_upgrade2018_realistic_v16_L1v1-v3+MINIAODSIM/8EDDE712-E106-7B48-AB0B-7E4580426E8E.root

Should we maybe talk with administrators of MIT submit50 and ask if they are interested in enabling tokens support?

[oshadura]

@jthiltges ah it will not help us, right? it looks like we need to support x509 at coffea-casa?