PersistentDict: Failure to recover RE.md from files on startup

[prjemian]

On starting a Bluesky session, a FileNotFoundError is raised from PersistentDict sometimes.

Expected Behavior

RE.md = PersistentDict(".RE_md")

should succeed.

Current Behavior

Sometimes, this fails with a FileNotFoundError exception.

Possible Solution

The reason why this happens only sometimes is not fully understood. Yet. In the case cited, the exception fails to find file ./versions%2330 (True, this file name does not exist). There is a different file, ./versions#30, that does exist. This represents backups of the RE.md["versions"] metadata key.

Why does PersistentDict look for a file with this other suffix?

Steps to Reproduce (for bugs)

See BCDA-APS/bluesky_training#300

Context

Routine beam line use is interrupted. User confidence in this aspect of Bluesky is compromised.

Your Environment

Linux (observed using various distributions including Ubuntu, Debian, RedHat)

[prjemian]

example

(bluesky_2024_2) prjemian@arf:~/Bluesky_RunEngine_md$ ipython
Python 3.11.8 | packaged by conda-forge | (main, Feb 16 2024, 20:53:32) [GCC 12.3.0]
Type 'copyright', 'credits' or 'license' for more information
IPython 8.22.2 -- An enhanced Interactive Python. Type '?' for help.

In [1]: from bluesky.utils import PersistentDict

In [2]: %xmode Verbose
Exception reporting mode: Verbose

In [3]: pd = PersistentDict(".")
---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
Cell In[3], line 1
----> 1 pd = PersistentDict(".")
        PersistentDict = <class 'bluesky.utils.PersistentDict'>

File ~/.conda/envs/bluesky_2024_2/lib/python3.11/site-packages/bluesky/utils/__init__.py:775, in PersistentDict.__init__(self=<PersistentDict {}>, directory='.')
    773 self._func = zict.Func(self._dump, self._load, self._file)
    774 self._cache = {}
--> 775 self.reload()
        self = <PersistentDict {}>
    777 # Similar to flush() or _do_update(), but without reference to self
    778 # to avoid circular reference preventing collection.
    779 # NOTE: This still doesn't guarantee call on delete or gc.collect()!
    780 #       Explicitly call flush() if immediate write to disk required.
    781 def finalize(zfile, cache, dump):

File ~/.conda/envs/bluesky_2024_2/lib/python3.11/site-packages/bluesky/utils/__init__.py:841, in PersistentDict.reload(self=<PersistentDict {}>)
    839 def reload(self):
    840     """Force a reload from disk, overwriting current cache"""
--> 841     self._cache = dict(self._func.items())
        self._cache = {}
        self = <PersistentDict {}>
        self._func = <Func: _dump<->_load <File: ., mode="a", 10 elements>>

File ~/.conda/envs/bluesky_2024_2/lib/python3.11/site-packages/zict/func.py:74, in <genexpr>(.0=<generator object ItemsView.__iter__>)
     73 def items(self) -> Iterator[tuple[KT, VT]]:  # type: ignore
---> 74     return ((k, self.load(v)) for k, v in self.d.items())
        self.load = <function PersistentDict._load at 0x7f21045d8ae0>
        self = <Func: _dump<->_load <File: ., mode="a", 10 elements>>
        self.d = <File: ., mode="a", 10 elements>

File <frozen _collections_abc>:861, in __iter__(self=ItemsView(<File: ., mode="a", 10 elements>))

File ~/.conda/envs/bluesky_2024_2/lib/python3.11/site-packages/zict/file.py:82, in File.__getitem__(self=<File: ., mode="a", 10 elements>, key='versions#30')
     80     raise KeyError(key)
     81 fn = os.path.join(self.directory, _safe_key(key))
---> 82 with open(fn, "rb") as fh:
        fn = './versions%2330'
     83     if self.memmap:
     84         return memoryview(mmap.mmap(fh.fileno(), 0, access=mmap.ACCESS_READ))

FileNotFoundError: [Errno 2] No such file or directory: './versions%2330'

In [4]: !ls
beamline_id#22	conda_prefix#24  databroker_catalog#23	iconfig#21  instrument_name#27	login_id#26  pid#28  proposal_id#29  scan_id#25  versions#30

In [5]: 

[prjemian]

Contents of this directory:
RE_md_persistent_dict.zip

[prjemian]

Problem appears to arise in upstream zict package:

In [11]: import zict

In [12]: zf = zict.File(".")

In [16]: dict(zf)
---------------------------------------------------------------------------
FileNotFoundError                         Traceback (most recent call last)
Cell In[16], line 1
----> 1 dict(zf)
        zf = <File: ., mode="a", 10 elements>

File ~/.conda/envs/bluesky_2024_2/lib/python3.11/site-packages/zict/file.py:82, in File.__getitem__(self=<File: ., mode="a", 10 elements>, key='versions#30')
     80     raise KeyError(key)
     81 fn = os.path.join(self.directory, _safe_key(key))
---> 82 with open(fn, "rb") as fh:
        fn = './versions%2330'
     83     if self.memmap:
     84         return memoryview(mmap.mmap(fh.fileno(), 0, access=mmap.ACCESS_READ))

FileNotFoundError: [Errno 2] No such file or directory: './versions%2330'

Same error as reported.

[prjemian]

Here's the troublemaker (called from zict/file.py line 81 above). I call it directly with the desired versions#30 file name:

In [23]: zict.file._safe_key("versions#30")
Out[23]: 'versions%2330'

In turn, this relies on urllib.parse.quote():

In [28]: from urllib.parse import quote

In [29]: quote("versions#30")
Out[29]: 'versions%2330'

where %23 is the rendering of #.

[prjemian]

Since this is a file name, not a URL, is urllib.parse.quote() being abused here?

The argument to PersistentDict() is named directory. It should not be cleaned up as if it were a URL. Different substitution rules apply. This is a case where the file names do not match.

[prjemian]

If we wrap this code with try..except FileNotFoundError: https://github.com/bluesky/bluesky/blob/add624e1b7f03c9b4b981e78dfc2333ed01e1f0c/src/bluesky/utils/__init__.py#L810C9-L811C1

What do we do in the exception handling? Report, then erase the directory? Seems brutal.

[prjemian]

A less brutal correction might be to remove the #* suffixes. That seems to to be the pattern that triggers the exception.

[prjemian]

Once the files ending with the suffix #n (where n is an integer) have been stripped of their suffixes, zict.File(".") opens the directory successfully.

In [5]: zf = zict.File(".")

In [6]: dict(zf)
Out[6]: 
{'instrument_name': b'\xbbBCDA EPICS Bluesky training',
 'pid': b'\xce\x00\x15\x82~',
 'conda_prefix': b'\xd9)/home/prjemian/.conda/envs/bluesky_2024_2',
 'login_id': b'\xb7prjemian@arf.jemian.org',
 'iconfig': b'\xde\x00\x1b\xb0ADSIM_IOC_PREFIX\xa3ad:\xadGP_IOC_PREFIX\xa3gp:\xb2DATABROKER_CATALOG\xa8training\xb2RUNENGINE_METADATA\x84\xabbeamline_id\xb0Bluesky_training\xafinstrument_name\xbbBCDA EPICS Bluesky training\xabproposal_id\xa8training\xb2databroker_catalog\xa8training\xb5RUN_ENGINE_SCAN_ID_PV\xabgp:gp:int20\xacAD_IMAGE_DIR\xb1adsimdet/%Y/%m/%d\xadAD_MOUNT_PATH\xa4/tmp\xb2BLUESKY_MOUNT_PATH\xb9/tmp/docker_ioc/iocad/tmp\xbaALLOW_AREA_DETECTOR_WARMUP\xc3\xd9!ENABLE_AREA_DETECTOR_IMAGE_PLUGIN\xc3\xacENABLE_CALCS\xc3\xb0USE_PROGRESS_BAR\xc2\xb6WRITE_NEXUS_DATA_FILES\xc2\xbaNEXUS_WARN_MISSING_CONTENT\xc2\xb4NEXUS_FILE_EXTENSION\xa2h5\xb5WRITE_SPEC_DATA_FILES\xc2\xb1RUNENGINE_MD_PATH\xd9+/home/prjemian/.config/Bluesky_RunEngine_md\xa7LOGGING\x81\xbaNUMBER_OF_PREVIOUS_BACKUPS\t\xafPV_READ_TIMEOUT\x0f\xb0PV_WRITE_TIMEOUT\x0f\xb5PV_CONNECTION_TIMEOUT\x0f\xb1XMODE_DEBUG_LEVEL\xa7Minimal\xb6MINIMUM_PYTHON_VERSION\x92\x03\x08\xb7MINIMUM_BLUESKY_VERSION\x92\x01\n\xb5MINIMUM_OPHYD_VERSION\x92\x01\x07\xbaMINIMUM_DATABROKER_VERSION\x92\x01\x02\xafICONFIG_VERSION\xa51.0.1',
 'beamline_id': b'\xb0Bluesky_training',
 'proposal_id': b'\xa8training',
 'databroker_catalog': b'\xa8training',
 'versions': b'\x8b\xa8apstools\xa61.6.18\xa7bluesky\xa81.13.0a3\xaadatabroker\xa51.2.5\xa5epics\xa53.5.2\xa4h5py\xa53.9.0\xa6intake\xa50.6.4\xaamatplotlib\xa53.8.4\xa5numpy\xa61.26.4\xa5ophyd\xa51.9.0\xabpyRestTable\xa82020.0.8\xaaspec2nexus\xa82021.2.6',
 'scan_id': b'\x00'}

[prjemian]

I'm suggesting that the caller can choose to clean up the directory before calling PersistentDict()directory). This problem will not need further handling in this code base.