Sync from Mac back to iPhone (Apple sync replacement) (Edited - now a library downsizer for iCloud solution)

[odedia]

Based on the discussion here, this feature request is to enable sync from macOS Photos Library to the iPhone, replacing Apple's limited functionality.
Currently, Apple sync is limited as follows:

• Converts all RAW files to 3mp jpegs
• Cannot convert unsupported video files to supported video files. Some of these files include regular .Mov files that are not exactly to spec according to Apple Devices requirements.
• Cannot sync back live photos.

The new feature should use something like iFuse or a third party library management on the iPhone to sync the library to the phone with all as much compatibility as possible.

Thanks!

[odedia]

Another option that came to mind is to have osxphotos create an "iCloud friendly" photos library from a "source of truth" library. It's a bit of a stretch so I'm just writing my thoughts below, I don't see this becoming a reality anytime soon.

This is the scenario:

I do love iCloud Photos Library, and I think it's probably the best cloud photo service out there. However - I'm at a point where my library is almost 2.3 terabytes. I treat my Photos Library as the "source of truth", which is why I store my RAW files in it as well. With my recent purchase of Nikon Z9, I no longer use Apple Photos to edit the RAW files (it can't open all permutations of the RAW file anyway), but I still keep the JPEG+RAW file combo for archiving and the potential of Apple getting their act together in the future and supporting the latest RAW formats.
Along with my wife and kids in a shared family account, we're close to 3 terabytes altogether for Photos, iCloud drive and iPhone backups.

Moving iPhone backups to backup to my Mac over WiFi is easy, and there are multiple iCloud Drive alternatives out there. But the main caveat that keeps me paying for iCloud is Photos. Since Apple doesn't offer all Apple One plans in my region, this means that there's no 4tb option (which is also expensive), and I must jump to the 6Tb a month plan, which is getting painful to say the least.

What would be ideal is something like this:

Have two Apple Photos Libraries on my Mac:

1. The "source of truth" 2.3Tb library, which is not backed up to iCloud. I would backup this library using Time Machine and also using osxphotos to a NAS, plus Backblaze.

• An iCloud Photos Library which would be a scaled down version of the source of truth.

The scaled-down version can be done in any number of ways: Strip the RAW files from JPEG+RAW combos, create smaller-sized JPEGs from the original RAWs or RAW+JPEG or full-sized JPEG, perhaps even transcoding videos at some point.
For reference - I checked my Photos library and it contains 1.1Tb of RAW files and 700gb of videos. The regular iPhone photos (with Live Photos) is only 120gb. That's with images dating back to iPhone 1. Converting the RAW files to mobile-friendly JPEGs would scale them down to around 70gb or less(!). This would also allow me to keep having live photos on my iPhone because Apple doesn't sync Live Photos back to an iPhone via regular sync, only via iCloud. I would easily be less than 2Tb of iCloud storage, or even 200gb if I don't sync Videos. I consider moving regular videos to a different service like Plex or Immich anyway.

For this to work, there needs to be some dual-library magic going on:

1. When I add videos or images from my mirrorless camera to the source-of-truth library, they should be scaled down, converted and synced to the iCloud Photos library. The schedule can be via a cronjob (or triggers if that's doable).
2. When I take a picture or video on my iPhone, that would upload to my iCloud Photos library. osxphotos would identify these photos that it doesn't currently have in the source-of-truth, and add them to the source-of-truth library. I don't think there's a reason to do any downscaling on iPhone images so these should just stay as-is in the iCloud library, but that's a potential for a future enhancement (it makes more sense if video downscaling is added at a later date).
3. If video conversion is supported - need to make sure that exif data is preserved in the target video. I tried converting some old MPEG2 videos to .MOV and it's surprisingly insistent on changing the metadata to date of conversion instead of original date (and geo) it was taken. If ffmpeg is the tool that is used to convert to the target iCloud library, this can support a ton of old obscure video formats such as .MPG, .DV, .MPEG2 and more.

Thanks again!

[RhetTbull]

This is definitely possible! I am working on a new core for OSXPhotos using the native Apple API. The public, documented API does not provide the needed functionality but it is clear the features I need are there because Photos can work with multiple libraries, etc. So I've done a lot of reverse engineering and I have most of the private API that Photos uses figured out and am working on putting the scaffolding in place to be able to use it. Currently, I have code that can create a new library, open multiple libraries at once and export from one library and import to another. This would make it entirely possible to do this. You could have rules as you've said, for example, --strip-live, --convert-to-heic (smaller than JPEG), --scale-video 720p etc.

This would likely need to go both ways -- if you shoot on iPhone, those photos would need to be moved to "source of truth" and then re-imported to the iCloud library with the appropriate transformations.

I started this because I want to implement #909 to add a command that lets you modify photos in place (for example, strip the live video) without losing metadata.

[odedia]

My goodness, this would be perfect!
I'm happy to help anyway I can (testing, debugging etc.). I not well experienced in Python (still a Spring/Java guy here), but happy to assist with easy and cumbersome tasks if you have any.

[RhetTbull]

I'll let you know when I have something that can be played with. Due to some major life events happening for me it will likely be some months before I can devote enough time to really get this ready. I'm in "maintenance mode" until then.

[odedia]

Fully understand, thanks again for doing this on your personal time.

[ulilicht]

Great discussion - I have a similar issue and wondering as well how I can keep my iCloud library small while maintaining a source of truth somewhere else. I really like the scenario described by RhetTbull (#1503 (comment))

Following this thread for now, happy to help testing as soon as there is something to test.