Ignore duplicates

[mikapietrus]

Hey,

Maybe it doesn't exist, or maybe I simply can't find it. I'm trying to import photos to album, but only new ones. I would like to ignore duplicates. So far I tried:

osxphotos import --album "2024/album_name" --split-folder "/" --dup-check /path/to/photos/*

However, it force to check duplicates and I can decide for each photo should I ignore. Problem is, that even when I check "Apply for duplicates" and click "Don't import" I need to do it for each photo. Which can be tricky if I'll need to upload 1000 photos.

Is there any way to do it? Maybe you could consider option like "--dup-ignore"?

[RhetTbull]

This is a good idea. This feature doesn't currently exist but should be fairly easy to implement (assuming photos are 100% identical). Photos computes a fingerprint of each photo and checks that for dupes on import so osxphotos can compute the same fingerprint and do the check.

[RhetTbull]

File size alone probably isn't good enough but I have figured out how to compute the same fingerprint Photos uses (it's hidden in a private API) so that should work. Will mean each file needs to be read in its entirety before import.

[mikapietrus]

If you would be able to add 2 options:
--dup-ignore - upload only photos with different fingerprint, ignore the ones with the same one. That will duplicate photos with the same filename
--dup-overwrite - upload photos with different fingerprint and overwrite with different fingerprint and the same filename

that would be useful combo

[RhetTbull]

The second is much trickier because filename alone isn't a good indicator that this is the "same" photo. osxphotos also cannot currently overwrite or delete a file in Photos. It could move files to a "To be deleted" album though. This might be a better option for a separate post processing script that keeps only the newest version of any photo with multiple copies with the same filename.

[mikapietrus]

I believe moving to "To be deleted" is still not bad solution

[RhetTbull]

I've implemented --skip-dups for osxphotos import and will include this in the next release.

[mikapietrus]

Wonderful! Can’t wait to test it!

…

On 23 Nov 2023, at 20:15, Rhet Turnbull ***@***.***> wrote: I've implemented --skip-dups for osxphotos import and will include this in the next release. — Reply to this email directly, view it on GitHub <#1262 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AO2PBI6BRMMFBXAKW77DON3YF6HCLAVCNFSM6AAAAAA6ZIWYG6VHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM3TMNJUGI2DE>. You are receiving this because you authored the thread.

[vodium-vastro]

Sorry to jump on this but feel it's related. I'm trying to do the same thing and looking to utilise --skip-dups. Quick question @RhetTbull, before I do it, does the --dry-run option also work for osxphotos import? I would like to be able to view what would be added so I can manually verify a few items first.

Many thanks

[RhetTbull]

Yes. See osxphotos help import or here

[vodium-vastro]

Perfect, thanks.

Is there any way to define what checks class a photo as duplicate? Seems my Google Takeout contains most images with same resolution, same file name and dates etc but file sizes are much smaller. I would like to class them as a dupe in favour of the higher resolution already in Apple Photos.

[RhetTbull]

@harveydov

Is there any way to define what checks class a photo as duplicate?

At the moment, there is not. This is what the code currently does:

If item is a photo and there is a fingerprint stored in the database the fingerprint of the photos is compared. In all but a small number of cases, Photos stores a unique fingerprint (a hash of the file) for each photo. osxphotos will call the same fingerprint method used by Photos to compare. Thus a match is only possible if both files are absolutely identical.

If item is a video or the database didn't contain a fingerprint, the code creates a fingerprint of filename + size of the image (in bytes) thus again, there needs to be an exact match. This second method is not as precise as the first and could result in false matches but Photos does not store a fingerprint for videos (I'm guessing because it's a slow operation to compute a fingerprint for a large file).

It would be possible to add a --fingerprint TEMPLATE option that allows you specify a custom fingerprint using the osxphotos template language. For example:

--fingerprint "{filepath.name}{photo.height}{photo.width}"

I will open a new issue for this. This will likely require the user install exiftool to use this as osxphotos would need a way to read the size and other metadata from the image prior to import and I don't want to write custom code to do this.

Note that using height and width would lead to greater likelihood of false matches as all images from a given camera/phone model will have the same height / width (unless cropped).

A templated fingerprint would allow you to use date instead of size or in addition, for example: --fingerprint "{filepath.name}{photo.height}{photo.width}{created}" (though Google mangles the dates by converting everything to UTC...)

[RhetTbull]

@harveydov The next release of osxphotos will include a --signature TEMPLATE option for the osxphotos import command that allows you to specific a custom signature template for use with --skup-dups. It will only provide a subset of the full template language (only those properties which can be determined via the file to be imported sidecar or exiftool metadata) but should be sufficient for doing what you suggested.