Sometimes have lots of "missing" photos in export (even with --download-missing)

[slobak]

I have used osxphotos to set up a rolling backup of a library of ~100k photos that only backs up a filtered set added to the library in the last few months (maybe it's 1-2k photos+vids). I use iCloud with "Optimize Mac Storage" so many of the originals are not stored locally until Photos has to fetch them, thus I am using the --download-missing flag. This means when I run the backup, Photos has to fetch the original file for many/most of the photos.

Note, I do not run with the flag --use-photokit because it causes a crash.

Here is what I've observed:

• Some of the photos still end up "missing" - this could be 10% (or more) of them in a run. It is often the same ones but not always! So it seems like some combination of the data itself and also a transient issue with either osxphotos or Photos.
• Sometimes my system gets in a state where all subsequently "exported" files are considered missing, like Photos stops trying to download anything. When this happens during an export, literally every remaining photo will be missing. Rebooting my system seems to fix this, until it happens again later.
• I know that Photos will sometimes refuse to download/sync if available hard drive space is too low (usually < 15GB or so?). I have ensured now that I run the backup with at least 50GB available so even if it downloads lots of data there is still plenty of headroom for its operation.

Any ideas what's going on?

I sometimes use the --add-missing-to-album for debugging, so I can see which images are missing and it updates in real-time. I also find if I click into the images then it forces Photos to sync them locally, and that may make it possible to export on the next run. But I can't do that with like 100 images, unless there's a way to easily automate Photos locally syncing an album? (e.g. via AppleScript).

[RhetTbull]

OSXPhotos really isn't designed to be used with an optimized library. With --download-missing it should theoretically work but that depends on the very fickle Photos AppleScript interface.

Sometimes my system gets in a state where all subsequently "exported" files are considered missing, like Photos stops trying to download anything.

That's likely due to the Photos AppleScript interface, which osxphotos uses to force the download. Sometimes then AppleScript interface crashes and there's nothing osxphotos can do to fix it. That is a bug in Photos. Apple cares very little about the AppleScript interface.

You said --use-photokit causes a crash. If we can get that working you'll find it much more reliable than AppleScript. This does require a different set of permissions which Terminal should have promoted you for. Verify in Settings > Privacy & Security that the terminal has permissions for the Photos library.

[RhetTbull]

Also, you might consider iCloud Photos Downloader which is designed to make a backup from iCloud instead of the local library.

[slobak]

Thank you this is helpful context, good to know the AppleScript interface is sketchy. I'll search/file a separate issue for the crash with --use-photokit.

[slobak]

Also iCloud Photos Downloader has waaaay fewer features than osxphotos! This is a really great and powerful library I'd love to continue to be able to use.