How to use osxphotos to recreate my library and preserve as much as possible?

[bernstbj]

My Photos library seems to have something wrong with it which neither the Photos.app itself nor the commercial PowerPhotos seems to be able to resolve. In general, it never finishes detecting faces after months of being left alone and even the photos I added a few weeks back still don't have face circles around them. Looking at the console I think things get stuck or daemons crash, but it might be things as stupid as the photoanalysisd trips on an AVI codec from 23 years ago that isn't recognized by the Mac.

So that has led me to doing a rebuild of my Photos library: export everything, take ancient videos and freshen them with HandBrakeCLI, convert some BMPs which are in there. I've exported the whole lot with osxphotos sorting the output with year/month/albumname and enabled --exiftools. Great, everything is out and I ran a script of mine that did the video upgrades and preserves the EXIF metadata, timestamps, etc.

So now that I have it all cleaned up, I'm looking to import into a newly created Photos.app database.

My questions:

1. Is there a way to import into a specified Photos library like there was during the export and the --db option? Doesn't seem like it.
2. Is there a way to recreate the albums and other folder organizations with the import function? Or is that something I'd have to script on my own to achieve.
3. Will the import function along with the --exiftool argument take all the metadata that was exported from the original run and put it back into this recreated library?
4. Is there a better way I should have exported the library with the goal of reimporting to rebuild and lose litte-to-no data?

If things like getting items into the right albums need to be scripted by me (e.g. find directory names that are not '_'), that's doable. However, if anyone else has used this software to do a minimal-loss rebuild, I'd be happy to hear!
Thanks

[RhetTbull]

Is there a way to import into a specified Photos library like there was during the export and the --db option? Doesn't seem like it.

The osxphotos import command doesn't currently support specifying the library to import to -- it uses the last opened library. So in this case you'd create a new library (quit Photos then open it while holding down Option) and run the import command. See the docs for import here.

Is there a way to recreate the albums and other folder organizations with the import function? Or is that something I'd have to script on my own to achieve.

Yes. Type osxphotos help import to see the full help for import. For example (from the help):

If your photos are organized in a set of folders but you want to exclude one or more parent folders from the
list of folders and album, you can use the --relative-to option to specify the parent path that all subsequent
paths should be relative to. For example, if your photos are organized into photos as follows:

 /                                                                                                            
 └── Volumes                                                                                                  
     └── Photos                                                                                               
         ├── 2021                                                                                             
         │   ├── Family                                                                                       
         │   └── Travel                                                                                       
         └── 2022                                                                                             
             ├── Family                                                                                       
             └── Travel                                                                                       

and you want to exclude /Volumes/Photos from the folder/album path, you can do this:

osxphotos import /Volumes/Photos/* --walk --album "{filepath.parent}" --relative-to "/Volumes/Photos"
--split-folder "/"

This will produce folders/albums 2021/Family, 2021/Travel, and so on.

Will the import function along with the --exiftool argument take all the metadata that was exported from the original run and put it back into this recreated library?

Yes, with exception of persons/faces.

Is there a better way I should have exported the library with the goal of reimporting to rebuild and lose little-to-no data?

Sounds like you did it right. The only option I might have done differently is to use --person-keyword to encode person names as keywords as person's cannot be currently imported. You'll have to re-tag all faces in either case but using the person name as keyword allows you use smart folders or search to find persons until they've been re-tagged. However, you can do this on import using the options: --keyword "{person}" --merge-keywords with your import command.

I think that with the options above, you should be able to recreate the album. Note however that this isn't really the intended use case for osxphotos import but I think it should work. The following command should be a good starting point (replace with your own paths):

osxphotos import /Volumes/Photos/* --walk --album "{filepath.parent}" --relative-to "/Volumes/Photos" --split-folder "/" --exiftool --keyword "{person}" --merge-keywords

I am working on some new code that will enable me to implement an osxphotos copy command (#938) and also an osxphotos transform (#909) that would allow you to re-code the videos in-place and update the Photos library with the new version (osxphotos transform) or copy assets with all metadata from one library to another (optionally creating a new library) (osxphotos copy). The latter may be helpful for cases where the Photos library is "corrupt" and Photos can't fix it but osxphotos can read it (because osxphotos doesn't rely on Photos' code to read the library) to create a new library. These are several months out but I'm making steady progress in between other projects.

[bernstbj]

This is useful information - thank you! I'm going to try the workflow again bit with the person keyword. Will see how this does on a scratch drive.

FWIW, I like how you're looking to bake in transcoding in a future release. I have a script I wrote a few years back that leverages HandBrakeCLI to do this, but re-adding to the photo library is a manual step. I was actually thinking of raising this as an enhancement ticket! That being said, a key thing I'd point out is that while older files tend to get a space savings by moving to something newer, it isn't always the case. The default behavior of HandBrake is to transcode to x264, and that usually will not do a better job than the HEVC / x265 codec that newer iPhones spit out (it WILL do better than my previous 6S or 4S). I haven't looked hard enough yet to figure out the magic combination of EXIF tags to determine if it isn't worth transcoding (Compressor ID / Name isn't definitive to the output), but in general it is a bit of a tradeoff of speed VS compatibility at the moment. I can go a bit more into this if you're interested.

[RhetTbull]

The default behavior of HandBrake is to transcode to x264, and that usually will not do a better job than the HEVC / x265 codec that newer iPhones spit out

It might be possible to code the transcoder directly as an osxphotos pllugin that calls the native encoder from AVFoundation....something to look at when I finish the current round of projects.

[oPromessa]

Hey osxphotos user community!

Recreating Library 100K referenced files

Piggy/backing on this thread I'm now considering recreating the 100K pics Library (coming from the iPhoto times and referenced photos/videos) to avoid/improve some current issues

• not all People analyzed
• the infamous 'Find Original...' message

BIG question: should I go import or sync say?

• Option A: export/import will place all files nicely into folder (my current folder structure I use for import is organised by date of import.. of course future imports once I recreate the Library will give me some work if I want to keep it!
• Option B: sync: would it be much faster than import?
  • I guess I still need to run osxphotos import prior to osxphotos sync
  • It will get all details from the library that I'd otherwise get via XMP.
  • Does it also get People and labels from current Library?

Option A: export/import option:

Some questions:

1. So will filter //live pics for now as import does not cater for them. To figure out later.
2. Exporting with --exiftool and --sidecar xmp and GPS and Keywords even on Movies seem to load/up nicely ;).
3. Importing with --sidecar xmp instead of --exiftoolasI'd export it to be faster to read XMP files than running exiftool on each file.
4. Question: What about Hidden pics, any way to export and import them hidden?
5. On the Edited pics front I'm going with --skip_original_if_edited and get cropped/rotated pics and loose Originals.
6. I think I saw some reference on a post (which I can't find now) on a --directory export dynamic template to achieve the following structure on export to then use on the import later. Is this possible?

2000-2002
      2000
          2000-02-03 Album Name 
          2000-07-07 Album Name 
          ...
      2001
          2001-02-03 Album Name 
          ...
      2002
          2003-09-03 Album Name 
          ...
2003-2005
      2003
          2003-03-15 Album Name 
          2003-07-07 Album Name 
          ...
      2004
          2004-12-03 Album Name 
          ...  

7. As to speed on SSD i7 iMac late 2015 (!) on Monterey (yes, can't upgrade :(. 200 pics/videos load in about 2 minutes. Very small test batch for now, I know !

Option B: sync: would it be much faster than import?

• Still not tested... will go at it now!

[RhetTbull]

@oPromessa Great! Thank you for taking time to test.

Quick question, do we still need to include --walk when specifying something like osxphotos import photos.hard/*

Yes, if you want osxphotos to also import files in subdirectories, you need to specify --walk. Without --walk, just specifying the directory will cause osxphotos to import all files in the top level of the specified directory but not those in the subdirectories. Rationale: User may want to only import a subset of photos without recursing through the whole tree. With the new change, osxphotos import photos.hard/ is equivalent to ``osxphotos import photos.hard/*`

Also, one other change I didn't mention is that osxphotos will now filter any files that are not image or video files. Previously it would try to import any file you specified which would cause Photos to generate an error if it wasn't a valid media file.

There is one other change I want to include before releasing this and that is to add the option to read from an osxphotos export database to reconstruct metadata such as albums. I think I've found a really elegant solution that won't require too much code change. I created a class that can reconstitute an osxphotos PhotoInfo object from the data stored in the database which allows me to just plug this in to the existing code.

[RhetTbull]

@oPromessa when you have time please take a look at the latest importapalooza branch. I believe this is now ready for testing an import of an exported library. A couple things to note:

For the export, you should use --export-aae if you want to preserve adjustments.

For import, try the following:

osxphotos import /path/to/export --walk --exportdb /path/to/export_or_exportdb --verbose

This will cause all metadata, including albums to be read from the export database. You can also use --sidecar if you used --sidecar xmp or --sidecar json on export. In this case, the data in the sidecar takes precedence over the export db (rationale: you might have edited the file/sidecar after export) but the sidecar does not store album info so you must still use --exportdb if you want to restore albums. Edits made to files in external editors will not be preserved -- only non-destructive edits made in Apple Photos are preserved in the AAE files. RAW+JPEG, Live, Bursts, should all be imported correctly.

If you use a different suffix for exporting the edited images, you'll also need to specify this with --edited-suffix. Edited images exported by osxphotos with the suffix will be automatically renamed on import. For example: IMG_1234.jpg and IMG_1234_edited.jpg will be renamed to IMG_1234.jpg and IMG_E1234.jpg to match the format expected by Photos for edited images. Anything after IMG_1234 will be preserved. If name is not in this format, it will be renamed to match the format with an incrementing counter, e.g. IMG_0001_original_name.jpg. This is because Photos requires this format for edited images. See comments on #1113

You can use --auto-live if you have live photos to import from Android or other platforms that aren't natively supported by Photos.

[oPromessa]

This is amazing. Will try to test it in the coming days.

[RhetTbull]

One note: to use the --exportdb option, the export must have been created with the version of osxphotos in this branch or you'll need to use python3 -m osxphotos exportdb --upgrade /path/to/export from the branch to update the exportdb with the needed changes.

[RhetTbull]

FYI, I just used this to export an old iPhoto library from 2014 (1104 photos) that I could no longer open then re-import into a new Photos library using osxphotos -- it worked great and reconstructed all metadata, albums, etc.