I did my best... And now I would be very grateful for your help!

[msg43]

I POSTED THIS TO R/OSXPHOTOS subreddit as well

Hi all!

My goal is to export my 200k photos/videos while saving every last little bit of ML brilliance Apple has imparted to my photos so I can more easily cull them and start putting them into scrap books for the grand parents...

I didn't want to ask super-dumb questions so I tried my best by studying the help files, reading the online discussions, googling errors, and asking ChatGPT for help. I started at zero knowledge and ended up here so far:

osxphotos export /Users/USER/Pictures/EXPORTFOLDER --filename "{created.date}-{counter:05d(1)}" --directory "{created.year}/{created.mm}" --exiftool --person-keyword --album-keyword --favorite-rating --ignore-date-modified --finder-tag-keywords --report export2.csv --overwrite --skip-live

I am stuck on a few things:

1. When I do my export I get

Processed: 45552 photos, exported: 297, missing: 45549, error: 0

I cannot for the life of me figure out why the library is "missing" 99% of its images. I have checked the Apple Photos library and the files sure seem to be there (full sized files, not just low-res previews). I have tried with three different apple photos libraries. Same issues. This photo library is 15GB.

2. I have also tried just running "% osxphotos export /Users/USER/Pictures/FOLDER" and I get the same result.

3. All of the 297 exported files in my small library end with "_edited".

4. The file number I added to the end (using --filename "{created.date}-{counter:05d(1)}") seems to include the missing files, so they look like "2013-01-08-45264_edited.jpeg" which is interesting because it seems like the counter increments even though the file is "missing". Not a problem per se, but annoying since it yields folders with essentially random file numbers.

5. Perhaps similar to Feature: --version to command line utility #4, I get a lot of folders which are empty. I guess it is creating the year folder and month folder and then going to find the photo itself and determining that it is "missing"?

6. The 15GB library was a test library to save time while I practiced. When that didn't work I decided to just try the command on my main library which is 700GB. With the same command I DID get lots of files, and those files were full size, and they were JPG, HEIC, and PNG, and those files did not say "_edited". HOWEVER, I get a zillion errors in the form of:

Error: Not a valid PNG (looks more like a JPEG)
or
Error: Not a valid HEIC (looks more like a JPEG)

I had to terminate the experiment because it would have run for hours spitting out these errors.

I found one person who had the same errors I had (I also used Google Takeout at one point). The gurus suggested that OP fix the problem by running exiftool. The post is at the following link.

Link: https://exiftool.org/forum/index.php?topic=12602.0

The problem is that I dont understand how to run exiftool inside osxphotos to achieve the outcome this post was suggesting. I tried to run it like this:

osxphotos export /Users/matthewgreer/Pictures/EXPORT9 --filename "{created.date}-{counter:05d(1)}" --directory "{created.year}/{created.mm}" --exiftool -ext heic '-Filename<$BaseName.FileTypeExtension' --person-keyword --album-keyword --favorite-rating --ignore-date-modified --finder-tag-keywords --report export2.csv --overwrite --skip-live

And I get the following error:
Error: No such option: -e

NB: https://www.reddit.com/r/osxphotos/comments/18uqm22/comment/kfmamp3/?utm_source=share&utm_medium=web3x&utm_name=web3xcss&utm_term=1&utm_content=share_button

THANK YOU SO MUCH TO EVERYONE FOR YOUR HELP AND PATIENCE!!

[RhetTbull]

Happy to help but need a bit more data.

I cannot for the life of me figure out why the library is "missing" 99% of its images.

Are you using iCloud Photos? If so, in Photos > Settings > iCloud > iCloud Photos is "Optimize Mac Storage" checked? If so, then the photos are missing because they're in iCloud but not downloaded to the Mac. They appear in Photos because Photos keeps a preview image then downloads them on demand.

If you are not using iCloud or "Download originals to this Mac" is checked, then something else is wrong and I'll provide some more debug steps.

The 15GB library was a test library to save time while I practiced.

So I'm guessing it's not an iCloud library. If so, how did you create it?

All of the 297 exported files in my small library end with "_edited".

This must have something to do with how the test library was created. Need more info on that.

The file number I added to the end (using --filename "{created.date}-{counter:05d(1)}") seems to include the missing files, so they look like "2013-01-08-45264_edited.jpeg"

The {counter} template increments each time it's evaluated and it's evaluated for every asset being checked for export. The rendered filename is produced before osxphotos knows if the file is present in the library to be exported or not so that's why you're seeing this. All the "missing" photos incremented the counter. The counter is probably not really what you want -- it's a dumb counter that simply increments and isn't unique to the photo. What are you trying to accomplish? Why are you appending the counter? You might prefer to use {id} which is a unique number for each photo. If you don't include a {counter} or {id} you'll automatically get 2023-01-08 (1).jpeg, 2023-01-08 (2).jpeg and so on.

2. HOWEVER, I get a zillion errors in the form of: Error: Not a valid PNG (looks more like a JPEG) or Error: Not a valid HEIC (looks more like a JPEG)

These errors are from exiftool and it appears you've got some images that have one extension but are actually something else. I've seen this before but usually related to editing images in a 3rd party app that didn't follow the guidelines for editing photos. It could also be related to importing photos from Google photos if something got mixed up in the process. When you see these errors there should be an error message that includes the UUID (universally unique identifier) for the photo. Given the UUID, you can do the following:

osxphotos debug-dump --dump photos --uuid UUID > debug.txt (substitute the UUID for UUID) Post the file here or send to me at osxphotos.py@gmail.com and I'll take a look to see what Photos says the file really is.

The problem is that I dont understand how to run exiftool inside osxphotos to achieve the outcome this post was suggesting. I tried to run it like this:

You cannot run exiftool this way. osxphotos only uses exiftool to output the metadata to the exported photo not to do any other transforms and not against the original image in the library. If it's an iCloud library doing so risks data corruption. It would be possible to run exiftool against the library using a custom script that uses osxphotos but I'd need to understand a lot more about the issue before even attempting something like this.

2. I get a lot of folders which are empty. I guess it is creating the year folder and month folder and then going to find the photo itself and determining that it is "missing"?

That's probably what's happening. It may be that osxphotos is creating the folder before it knows if the file is actually not missing and thus leaving an empty folder. If so, this might be classified as a bug. I'll open a separate issue for this so I remember to look into it. If you include the --cleanup option to your export command, it will cleanup (delete) any files and folders that osxphotos didn't export and that should clean up these ghost directories. But definitely something I'll look into.

My goal is to export my 200k photos/videos while saving every last little bit of ML brilliance Apple has imparted to my photos

Your export command won't do this. In particular, the machine learning labels ("dog", "tree", "beach", etc.) that Photos adds to help with searching won't be exported. OSXPhotos calls these labels and you can include these if you like by adding --keyword-template "{label}" to your export command.

[RhetTbull]

I opened #1410 for the bug related to empty directories.

[msg43]

Thank you so so much for your help. My comments below inline with yours!

On Tue, Feb 20, 2024 at 11:29 PM Rhet Turnbull ***@***.***> wrote: Happy to help but need a bit more data. I cannot for the life of me figure out why the library is "missing" 99% of its images. Are you using iCloud Photos? If so, in Photos > Settings > iCloud > iCloud Photos is "Optimize Mac Storage" checked? If so, then the photos are missing because they're in iCloud but not downloaded to the Mac. They appear in Photos because Photos keeps a preview image then downloads them on demand.

I started with an icloud library. I always keep my library on "download originals" not "optimize space". That original library was 1-0. I made changes to that library using powerphotos to merge other smaller libraries like the "test" library into the main library. As I did so I changed the file number to 1-1, 1-2, 1-3, and now 1-4. None of those versions have been connected to the internet. Based on your concern I have made 1-4 the system library and selected "download all photos". As you know I can't control the sync process but in the next 24 hours I should be able to confirm that there are no files stranded in the cloud.

If you are not using iCloud or "Download originals to this Mac" is checked, then something else is wrong and I'll provide some more debug steps. The 15GB library was a test library to save time while I practiced. So I'm guessing it's not an iCloud library. If so, how did you create it?

The test library is an example of a library I created by gathering files from other sources with a goal of unifying all the files in Apple Photos and then ultimately in Lightroom Classic. These files came from Google Photos. Others have been scanned in from paper photos, etc.

All of the 297 exported files in my small library end with "_edited". This must have something to do with how the test library was created. Need more info on that.

The test library is a subset of my photos. I'm guessing that out of the 45552 photos, 297 photos had been lightly edited in some way. Other than that I am not sure what I did while creating that library which was out of the ordinary. As I mentioned above I HAVE been slamming libraries together and pulling them apart and transferring them back and forth from NAS drives so it is conceivable that in the process I broke something inside the Photos database(?).

The file number I added to the end (using --filename "{created.date}-{counter:05d(1)}") seems to include the missing files, so they look like "2013-01-08-45264_edited.jpeg" The {counter} template increments each time it's evaluated and it's evaluated for every asset being checked for export. The rendered filename is produced *before* osxphotos knows if the file is present in the library to be exported or not so that's why you're seeing this. All the "missing" photos incremented the counter. The counter is probably not really what you want -- it's a dumb counter that simply increments and isn't unique to the photo. What are you trying to accomplish? Why are you appending the counter? You might prefer to use {id} which *is* a unique number for each photo. If you don't include a {counter} or {id} you'll automatically get 2023-01-08 (1).jpeg, 2023-01-08 (2).jpeg and so on.

Thanks for your guidance on this. I am trying to have a folder hierarchy YEAR > MONTH and fiile name YYYY-MM-DD-TOPIC-#### where each topic (i.e. xmas) starts over from 0001 (I think? Tell me if you disagree with that plan!). Topics were going to be pulled from Album Names.

1. HOWEVER, I get a zillion errors in the form of: Error: Not a valid PNG (looks more like a JPEG) or Error: Not a valid HEIC (looks more like a JPEG) These errors are from exiftool and it appears you've got some images that have one extension but are actually something else. I've seen this before but usually related to editing images in a 3rd party app that didn't follow the guidelines for editing photos. It could also be related to importing photos from Google photos if something got mixed up in the process. When you see these errors there should be an error message that includes the UUID (universally unique identifier) for the photo. Given the UUID, you can do the following: osxphotos debug-dump --dump photos --uuid UUID > debug.txt (substitute the UUID for UUID) Post the file here or send to me at ***@***.*** and I'll take a look to see what Photos says the file really is. Do you want me to get all the UUIDs for all the error photos or just a

single representative photo? It seems like they are all either "Not PNG" or "Not HEIC" so I could get one of each?

The problem is that I dont understand how to run exiftool inside osxphotos to achieve the outcome this post was suggesting. I tried to run it like this: You cannot run exiftool this way. osxphotos only uses exiftool to output the metadata to the exported photo not to do any other transforms and not against the original image in the library. If it's an iCloud library doing so risks data corruption. It would be possible to run exiftool against the library using a custom script that uses osxphotos but I'd need to understand a lot more about the issue before even attempting something like this. 1. I get a lot of folders which are empty. I guess it is creating the year folder and month folder and then going to find the photo itself and determining that it is "missing"? That's probably what's happening. It may be that osxphotos is creating the folder before it knows if the file is actually not missing and thus leaving an empty folder. If so, this might be classified as a bug. I'll open a separate issue for this so I remember to look into it. If you include the --cleanup option to your export command, it will cleanup (delete) any files and folders that osxphotos didn't export and that should clean up these ghost directories. But definitely something I'll look into. Great, thanks for explaining that. Will do. My goal is to export my 200k photos/videos while saving every last little bit of ML brilliance Apple has imparted to my photos Your export command won't do this. In particular, the machine learning labels ("dog", "tree", "beach", etc.) that Photos adds to help with searching won't be exported. OSXPhotos calls these labels and you can include these if you like by adding --keyword-template "{label}" to your export command.

Ok, I'm glad you brought that up since my MAIN reason for using OSXPhotos is to grab all these goodies like apple's hidden keywords, aesthetic score, face info, finder tags, etc. because it doesn't seem like any of the LRC plug-ins or stand-alone apps seem to do that great a job at ML keywording. I started with a more complicated export command which included --keyword-template "{label}". For instance my export command was: osxphotos export --export-by-date --db=/Users/matthewgreer/Pictures\ Family Photo Database.photoslibrary --filename=%Y-%m-%d-%04d --skip-live --exiftool-option -m --favorite-rating --ignore-date-modified --person-keyword --keyword-template "{folder_album(>)}" --keyword-template "{label}" --finder-tag-keywords --xattr-template findercomment "{title,}{title?{descr?{newline},},}{descr,}" --dry-run --verbose --report export1.csv --overwrite Users/matthewgreer/Pictures/EXPORT1 --retry 3 But I get this response: Error: No such option: --keyword-template {label} Did you mean --keyword-template? *Can I send you money on buy me a coffee or patron or something? * THANKS! Matt —

…

Reply to this email directly, view it on GitHub <#1409 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AR6K5LIUAOLWB4IPSFJEQZDYUVZZ3AVCNFSM6AAAAABDSF5S62VHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4DKMZYGM4DK> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[RhetTbull]

I started with an icloud library. I always keep my library on "download
originals" not "optimize space". That original library was 1-0. I made
...Based on your concern I have made 1-4 the

As I mentioned above I HAVE been slamming libraries together
and pulling them apart and transferring them back and forth from NAS drives
so it is conceivable that in the process I broke something inside the
Photos database(?).

I'm guessing something happened in this process that caused the library to not be the system library (so it wasn't downloaded from iCloud) but all the images had not yet been downloaded from iCloud before this happened. With as many changes to libraries as it sounds you've been making it is hard to know what state the library is actually in.

Thanks for your guidance on this. I am trying to have a folder hierarchy
YEAR > MONTH and fiile name YYYY-MM-DD-TOPIC-#### where each topic (i.e.
xmas) starts over from 0001 (I think? Tell me if you disagree with that
plan!). Topics were going to be pulled from Album Names.

This isn't possible currently given your use case. There is an {album_seq} (or {album_seq(1)} to start from 1 instead of 0) template that gives the sequence number for the photo in the album (album sort order). For example {album_seq:04d} would give you 0000, 0001, etc. However currently this only works if the album name is used in the --directory template. The rationale for this is that a photo can be in multiple albums and so osxphotos needs to know which album are we referring to in order to compute the sequence number. The most common use case is that people export to folders that match the album name thus osxphotos can inspect the folder template to see what album to look up. In your case, with album in the filename but not the folder name, I'd have to change the code to look at the filename template. This is probably worth doing so I'll open a new issue to add this.

If this was implemented you could do something like this:

--filename "{created.date}-{album?{album}-{album_seq(1):04d},NoAlbum-{id}}"

which would use AlbumName-0001 and so on for photos in an album but NoAlbum-ID for photos not in an album.

Do you want me to get all the UUIDs for all the error photos or just a
single representative photo? It seems like they are all either "Not PNG"
or "Not HEIC" so I could get one of each?

Just pick one of each PNG/HEIC and send the data.

Ok, I'm glad you brought that up since my MAIN reason for using OSXPhotos
is to grab all these goodies like apple's hidden keywords, aesthetic score,
face info, finder tags, etc. because it doesn't seem like any of the LRC
plug-ins or stand-alone apps seem to do that great a job at ML keywording.

You can easily get labels (hidden keywords) using --keyword-template "{label}", the persons using --person-keyword

You've already figured out Finder tags with --finder-tag-keywords and --xattr-template

The faces will be written to the XMP data using --exiftool or --sidecar XMP

The aesthetic score though is bit tougher. That's not exported by default and I'm not sure where you'd stick it as there's no standard for this kind of metadata. One option might be to append it to the description:

--description-template "{descr?{descr}{newline},}Score:{format:float:0.2f,{photo.score.overall}}"

This sets the description (caption) of the photo to the existing description if there is one, followed by a newline then adds "Score:0.75", etc.

I started with a more complicated export command which included
But I get this response: Error: No such option: --keyword-template {label}
Did you mean --keyword-template?

Odd. I can't reproduce this. It sounds like you're missing a space between --keyword-template and {label}. If writing commands in something like TextEdit or Notes be very wary of smart quotes and other auto-correct shenanigans which will mess up the command.

*Can I send you money on buy me a coffee or patron or something? *

I appreciate the sentiment but for various reasons I don't accept donations for osxphotos. Thanks though!

[msg43]

I started with an icloud library. I always keep my library on "download
originals" not "optimize space". That original library was 1-0. I made
...Based on your concern I have made 1-4 the

As I mentioned above I HAVE been slamming libraries together
and pulling them apart and transferring them back and forth from NAS drives
so it is conceivable that in the process I broke something inside the
Photos database(?).

I'm guessing something happened in this process that caused the library to not be the system library (so it wasn't downloaded from iCloud) but all the images had not yet been downloaded from iCloud before this happened. With as many changes to libraries as it sounds you've been making it is hard to know what state the library is actually in.

I have set 1-4 to sync w icloud. Should this theorectically fix the problem? AP works at its own mysterious pace. Would it be faster to run "--download-missing" ?

Thanks for your guidance on this. I am trying to have a folder hierarchy
YEAR > MONTH and fiile name YYYY-MM-DD-TOPIC-#### where each topic (i.e.
xmas) starts over from 0001 (I think? Tell me if you disagree with that
plan!). Topics were going to be pulled from Album Names.

This isn't possible currently given your use case. There is an {album_seq} (or {album_seq(1)} to start from 1 instead of 0) template that gives the sequence number for the photo in the album (album sort order). For example {album_seq:04d} would give you 0000, 0001, etc. However currently this only works if the album name is used in the --directory template. The rationale for this is that a photo can be in multiple albums and so osxphotos needs to know which album are we referring to in order to compute the sequence number. The most common use case is that people export to folders that match the album name thus osxphotos can inspect the folder template to see what album to look up. In your case, with album in the filename but not the folder name, I'd have to change the code to look at the filename template. This is probably worth doing so I'll open a new issue to add this.

If this was implemented you could do something like this:

--filename "{created.date}-{album?{album}-{album_seq(1):04d},NoAlbum-{id}}"

which would use AlbumName-0001 and so on for photos in an album but NoAlbum-ID for photos not in an album.

It would be super cool if you could. Although if it is too much work I guess I could export to folders with Album names. I have been trying to follow all these "family photo archiver" experts who seem to agree that one should follow a folder by YEAR > MONTH convention. But there is a small part of me that wonders if I should be foldering by ALBUM. But I think what they would say is that you just whip up a smart collection in LRC using your file names or keywords. <>

Do you want me to get all the UUIDs for all the error photos or just a
single representative photo? It seems like they are all either "Not PNG"
or "Not HEIC" so I could get one of each?

Just pick one of each PNG/HEIC and send the data.

I am sending you four TXT files by email (I can post them here if you prefer but I fear they have personal info in them).

One is a TXT file called "NOT A VALID FILE". This is a list of just the error output related to the "Not a valid XXXX (looks more like a JPEG)". Note there are more versions of this error than I originally realized...such as "Not a valid JPEG (looks more like a MOV)" etc... In total only about 1300 errors (mostly minor) for 200,000 files which seems pretty good (?).

One is a TXT file called "OTHER ERRORS" which is just a list of all the other errors (just in case thats helpful). You will see in that file that there are other errors like "atom is too large for rewriting (395 MB)" and "Error: Truncated 'e\xb4?\x95' atom - /var..."

One is the debug file for HEIC. One small caveat: when I ran the command "osxphotos debug-dump --dump photos --uuid 24FF9F4A-20AC-4162-901D-BE114B4957E8 > debug.txt" I got a long response in the terminal window but nothing in the output file so I copied what was in the window and pasted it into a txt file called DEBUG-HEIC.

One is the debug file for PNG. Same caveat so I copied what was in the window and pasted it into a txt file called DEBUG-PNG.

Ok, I'm glad you brought that up since my MAIN reason for using OSXPhotos
is to grab all these goodies like apple's hidden keywords, aesthetic score,
face info, finder tags, etc. because it doesn't seem like any of the LRC
plug-ins or stand-alone apps seem to do that great a job at ML keywording.

You can easily get labels (hidden keywords) using --keyword-template "{label}", the persons using --person-keyword

You've already figured out Finder tags with --finder-tag-keywords and --xattr-template

The faces will be written to the XMP data using --exiftool or --sidecar XMP

The aesthetic score though is bit tougher. That's not exported by default and I'm not sure where you'd stick it as there's no standard for this kind of metadata. One option might be to append it to the description:

--description-template "{descr?{descr}{newline},}Score:{format:float:0.2f,{photo.score.overall}}"

This sets the description (caption) of the photo to the existing description if there is one, followed by a newline then adds "Score:0.75", etc.

OK cool, thanks for the tip!

I started with a more complicated export command which included
But I get this response: Error: No such option: --keyword-template {label}
Did you mean --keyword-template?

Odd. I can't reproduce this. It sounds like you're missing a space between --keyword-template and {label}. If writing commands in something like TextEdit or Notes be very wary of smart quotes and other auto-correct shenanigans which will mess up the command.

I will send a screen capture in the email which I believe shows that I am not missing the space. But I am sure it is an "error between keyboard and chair" somehow. :)

*Can I send you money on buy me a coffee or patron or something? *

I appreciate the sentiment but for various reasons I don't accept donations for osxphotos. Thanks though!

OK, can I make a donation somewhere in your honor? :)

[RhetTbull]

Got the data and screenshot. Regarding the "No such option" error, I could not reproduce this. There were a couple of errors in the screenshot though. 1) You cannot use a template like --filename=%Y-%m-%-%04d, you must use the osxphotos template language. You could do this --filename "{created.date}" or you could use --filename="{created.strftime,%Y-%m-%d}" if you really wanted to use the strftime format codes. It also looked like you were missing the --exiftool option which is required if using --exiftool-option. I cannot tell from the screenshot where the error is but there's something amiss in your command line if you're getting the "No such option error" because this tells me osxphotos isn't parsing the command line the way you think it is. Here's a modified example based on your screenshot (with paths changed so I could test it) that does work:

osxphotos export --export-by-date --db ~/Pictures/Test-13.0.0.photoslibrary/ --filename="{created.date}-{id}" --skip-live --exiftool --exiftool-option -m --favorite-rating --ignore-date-modified --person-keyword --keyword-template "{folder_album(>)}" --keyword-template "{label}" --finder-tag-keywords --xattr-template findercomment "{title,}{title?{descr?{newline},},}{descr,}" --dry-run --verbose --report export1.csv --overwrite --retry 3 ~/Desktop/export

As for the "Not a valid file" errors, I trust exiftool more than I trust Photos so if exiftool reports the files are wrong, I believe it. I noticed both of the photos you sent debug data for were imported by PowerPhotos and one appeared to also have been edited by Google (perhaps Google Takeout?). I obviously can't provide support for those apps and I have no idea what path those photos took to end up in the library. They are clearly marked in the database as HEIC and PNG respectively but appear to be incorrect. If there's one of these that you can email me I can take a look to verify but again, if exiftool says it's wrong, I believe exiftool.

[msg43]

I am having a lot of fun working with this program! I have things mostly working! With regards to the "not a valid file" errors, StarGeek says in this thread to run a specific exiftool command to correct the extensions:

exiftool -ext heic '-Filename<$BaseName.FileTypeExtension' /path/to/files

Since I want to run these exiftool commands on a photo library I have tried to integrate the command by updating my command to read:

osxphotos export --retry 3 --library /Users/matthewgreer/Pictures/MG\ Photos\ 1-7.photoslibrary --directory "{created.year}" --from-date "2019-01-01" --to-date "2019-12-31" /Users/matthewgreer/Pictures/EXPORT20 --filename "{function:/Users/matthewgreer/bin/album_sequence.py::album_seq(NOALBUM,1)}" --skip-live --exiftool -ext heic '-Filename<$BaseName.FileTypeExtension' --exiftool-option -m --favorite-rating --ignore-date-modified --person-keyword --keyword-template "{folder_album(>)}" --keyword-template {original_name} --keyword-template "{label}" --finder-tag-keywords --dry-run --xattr-template findercomment "{title,}{title?{descr?{newline},},}{descr,}" --report /Users/matthewgreer/Pictures/export20.csv
but I get error Error: No such option: -e

if I pull out the -ext heic '-Filename<$BaseName.FileTypeExtension' the rest of the command runs just fine.

If I try just running exiftool -ext heic '-Filename<$BaseName.FileTypeExtension'/Users/matthewgreer/Pictures/MG\ Photos\ 1-7.photoslibrary I get No file specified

Any way I can accomplish my goals here?

THANKS!!!!

[msg43]

I also have an "Error: '0x0d0d0d0d' atom is too large for rewriting problem which Stargeek addresses in this thread.

Also any chance I can run FFMPEG on a photo library? (By using OSXPhotos or anything else)?

If not then I suppose I will have to export all the videos in the whole library, FFMPEG them, and then re-import them? Yikes...

Thanks Again!!

[RhetTbull]

if I pull out the -ext heic '-Filename<$BaseName.FileTypeExtension' the rest of the command runs just fine.

The command you listed is an exiftool command not an osxphotos command. You cannot just insert arbitrary exiftool commands into the osxphotos command line and expect it to work -- it's a different app. When you type --exiftool into the osxphotos command line, it tells osxphotos to use exiftool to write metadata to the exported files. OSXPhotos does this using a special set of exiftool commands but you cannot then add more commands on to run your own. You can pass options such as -m to ignore minor warnings via the --exiftool-option flag but that only works for options not entire commands.

I do NOT recommend running the exiftool command on your photos library as this might rename or replace files then neither Photos nor OSXPhotos will be able to find them.

How many of these problem files do you have?

You could run an exiftool command on the files post-export but that's a question for the exiftool forums, not here. I tried the command you posted and it did not appear to work correctly (I got some warnings about unknown tags):

exiftool -ext heic '-Filename<$BaseName.FileTypeExtension' /path/to/export

Warning: No writable tags set from /Users/rhet/Desktop/export/IMG_3092.heic
Warning: Invalid tag name '.filetypeextension'. Use '=' not '<' to assign a tag value - /Users/rhet/Desktop/export/IMG_3092.heic
Warning: No writable tags set from /Users/rhet/Desktop/export/IMG_3092_edited.heic
Warning: Invalid tag name '.filetypeextension'. Use '=' not '<' to assign a tag value - /Users/rhet/Desktop/export/IMG_3092_edited.heic
    1 directories scanned
    0 image files updated
    2 image files unchanged

I do have an open issue to apply transforms (such as running ffmpeg) on photos in the library then replacing the originals with the transformed image while preserving metadata. I've made some progress on the code needed to implement this but it's been on my back burner due to other priorities. See #909, #382

[RhetTbull]

@msg43 I took a look at implementing the ability to use {album_seq} in the --filename template and it's just not too difficult with the design of the current template system which operates in a context free way. Maybe someday I'll add context to the template system but that's a big redesign effort I'm not sure is worth the effort. In the meantime it occurred to me that we could use a {function} template to call a custom python function to achieve the same result. I've added a new example here that I think does what you want. If you save this as album_sequence.py in the same folder you run osxphotos from, you can the use this as your filename template:

--filename "{function:album_sequence.py::album_seq(None,1)}"

This will render filenames as YYYY-MM-DD-Album-Sequence starting from 1. If photo is not in an album it will be named YYYY-MM-DD-None-ID where None is the text used as the default above and ID is the row ID of the photo in the database. If you wanted to use "NoAlbum" and count from zero instead, you'd do this:

--filename "{function:album_sequence.py::album_seq(NoAlbum,0)}"

Alternatively, osxphotos will download the template function for you so you could call it like this:

--filename "{function:https://raw.githubusercontent.com/RhetTbull/osxphotos/main/examples/album_sequence.py::album_seq(None,1)}"

You can examine the source code of the function and change the template format to something else if desired.

[msg43]

In what order does the code process photos in an album for purposes of determining {album_seq} output number?

My situation:

When I run --filename "{function:album_sequence.py::album_seq(NoAlbum,0001)}" on a certain time period of files and some of those files are in a very large album that spans beyond the time period chosen then it seems like the code will {album_seq} all the files in the album and only export those in the date range which results in those files not starting from 0001 in their new exported home. They might instead start right in the middle of a number range. Does that sound right? It's not necessarily a problem, but I wanted to know if the next time I go to export and I have added photos to that album, will the code process the photos in the album in the same order and put those new photos at the end?

[RhetTbull]

it seems like the code will {album_seq} all the files in the album and only export those in the date range which results in those files not starting from 0001 in their new exported home.

Yes, that's the expected behavior. The {album_seq} sequence number is based on the asset's sequence in the album (the sort order that determines how the photos show up when you open the album). So if your filter criteria starts in the middle of the album, the {album_seq} will start in the middle of the album. E.g. if your export begins at photo # 50 then the {album_seq} will be 50.

[msg43]

I wanted to change the album_sequence.py code to output a file number which always started with 0001 instead of 1 without asking you for help.

I read in the help files under album_seq "to format while also using a starting value: '{album_seq:05d(1)}' which results in 0001, 00002...etc."

First I tried changing

--filename "{function:https://raw.githubusercontent.com/RhetTbull/osxphotos/main/examples/album_sequence.py::album_seq(None,1)}"
to
--filename "{function:https://raw.githubusercontent.com/RhetTbull/osxphotos/main/examples/album_sequence.py::album_seq:05d(1)}"

but I got the same output of YYYY-MM-DD-ALBUM-1 instead of YYYY-MM-DD-ALBUM-00001

Then I got ambitious and tried copying the raw text to a text file and putting it in the same folder as shown in the attached picture.

I ran the following with no changes to the code you wrote in album_sequence.py:

osxphotos export --directory "{created.year}" --from-date "1985-01-01" --to-date "1990-01-01" /Users/matthewgreer/Pictures/EXPORT9 --filename "{function:album_sequence.py::album_seq:05d(1)}" --skip-live --exiftool --exiftool-option -m --favorite-rating --ignore-date-modified --person-keyword --keyword-template "{folder_album(>)}" --keyword-template {original_name} --keyword-template "{label}" --finder-tag-keywords --xattr-template findercomment "{title,}{title?{descr?{newline},},}{descr,}" --dry-run --report /Users/matthewgreer/Pictures/export13.csv

but I got

'Error: Invalid value for '--filename': 'album_sequence.py' does not appear to be a file'

[msg43]

running % pipx list returns the results:

venvs are in /Users/matthewgreer/Library/Application Support/pipx/venvs
apps are exposed on your $PATH at /Users/matthewgreer/.local/bin
manual pages are exposed at /Users/matthewgreer/.local/share/man
package osxphotos 0.67.3, installed using Python 3.12.2

[RhetTbull]

You need to specify the full path to the album_sequence.py file or run the command from the directory where the file is located.

To get the formatting you want, you need to change the script. You cannot do it with formatting codes in the --filename template because you're using a custom template function. Change the line that says value += f"{album}-{seq}" to value += f"{album}-{seq:05d}"

[RhetTbull]

@msg43 osxphotos version 0.67.5 now includes a new osxphotos compare command that might be helpful in your situation. For example:

osxphotos compare Library1.photoslibrary Library2.photoslibrary --verbose --csv --output lib1_lib2.csv

will give you a CSV file listing every asset in both libraries and indicating if they're in the other, if they are the same as the other library, or if they're different (e.g. different metadata) and if so, what the differences are.

It is a resource hog as both libraries get loaded into memory so comparing two big libraries will be slow but this may help determine what's different in your multiple libraries.

[msg43]

This is so awesome thank you!