Does exiv2 read the whole file when printing EXIF tags?

[GabZach]

Well, I am wondering if exiv2 reads the whole file in case I issue the command exiv2 file.jpg ?

I am not talking about decoding the image, I am just wondering if exiv2 is clever enough to seek to the EXIF dictionary and read only that, instead of the whole file.

All insights will be appreciated!

Best regards, Gabriel

[clanmills]

Yes. Exiv2 uses an i/o object (which seeks appropriately), or memory mapping to minimize I/O. This is discussed in detail in my book: "Image Metadata and Exiv2 Architecture". https://exiv2.org/book/index.html

[clanmills]

@GabZach. That is exactly what exiv2 does! It understands the image formats and navigates with minimal effort to the EXIF/XMP/ICC/IPTC data. All other data is ignored. Exiv2 doesn't know anything about pixels and ignores them all. Allow me to provide an example for JPEG files which contain a linked list of "segments". The code "runs" the list searching for the APP1 segment containing the Exif date, APP1 containing XMP, App13 containing IPTC and APP1 containing ICC profile. The data is decoded into STL data structures. The scan terminates at SOS (Start of Scan)

$ 960 rmills@rmillsm1:~/Downloads $ exiv2 -pS https://clanmills.com/Stonehenge.jpg
STRUCTURE OF JPEG FILE: https://clanmills.com/Stonehenge.jpg
 address | marker       |  length | data
       0 | 0xffd8 SOI  
       2 | 0xffe1 APP1  |   15272 | Exif..II*......................
   15276 | 0xffe1 APP1  |    2786 | http://ns.adobe.com/xap/1.0/.<?x
   18064 | 0xffed APP13 |      96 | Photoshop 3.0.8BIM.......'.....
   18162 | 0xffe2 APP2  |    4094 | MPF.II*...............0100.....
   22258 | 0xffdb DQT   |     132 
   22392 | 0xffc0 SOF0  |      17 
   22411 | 0xffc4 DHT   |     418 
   22831 | 0xffda SOS  
961 rmills@rmillsm1:~/Downloads $ 

In "Stonehenge.jpg", Exiv2 has only read the first 22821 bytes of a 6.7MB file:

961 rmills@rmillsm1:~/Downloads $ curl -O https://clanmills.com/Stonehenge.jpg
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 6599k  100 6599k    0     0   655k      0  0:00:10  0:00:10 --:--:-- 1255k
962 rmills@rmillsm1:~/Downloads $ ls -l Stonehenge.jpg 
-rw-r--r--@ 1 rmills  staff  6757987 12 Jun 19:37 Stonehenge.jpg
963 rmills@rmillsm1:~/Downloads $ 

A similar strategy is applied for the 18 file formats (PNG, TIFF, GIF, CRW etc.) supported by Exiv2.

The code reads all four metadata standards (Exif, IPTC etc..). When you "open a file" Image::readMetadata() reads all four simultaneously. The effort required to do this is small. You can see reading the metadata over https is 10x faster than copying the image:

963 rmills@rmillsm1:~/Downloads $ time curl -O https://clanmills.com/Stonehenge.jpg
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 6599k  100 6599k    0     0   632k      0  0:00:10  0:00:10 --:--:-- 1164k
real	0m10.468s
964 rmills@rmillsm1:~/Downloads $ time exiv2 -g datetime/i https://clanmills.com/Stonehenge.jpg
Exif.Image.DateTime                          Ascii      20  2015:07:16 20:25:28
Exif.Photo.DateTimeOriginal                  Ascii      20  2015:07:16 15:38:54
Exif.Photo.DateTimeDigitized                 Ascii      20  2015:07:16 15:38:54
real	0m1.630s
965 rmills@rmillsm1:~/Downloads $ 

[clanmills]

Exiv2 is fast. On my 2 year old MacBookPro with the M1 chip, it reads about 1000 images/second. I had a trip at Easter and took 670 photos:

983 rmills@rmillsm1:~/Pictures/Photos/2023/Trips/Easter $ find . -name "*.jpg" | wc
     670     670   16174
984 rmills@rmillsm1:~/Pictures/Photos/2023/Trips/Easter $ time find . -print0 -name "*.jpg" | xargs -0 exiv2 -g 
     670    4020   70588
real	0m0.634s
985 rmills@rmillsm1:~/Pictures/Photos/2023/Trips/Easter $ time find . -print0 -name "*.jpg" | xargs -0 exiv2 -g Exif.Photo.DateTimeOrig 2>/dev/null | head -3
./Stirling/dsc_3317.jpg  Exif.Photo.DateTimeOriginal                  Ascii      20  2023:03:24 14:03:40
./Stirling/dsc_3303.jpg  Exif.Photo.DateTimeOriginal                  Ascii      20  2023:03:24 13:53:51
./Stirling/dsc_3288.jpg  Exif.Photo.DateTimeOriginal                  Ascii      20  2023:03:24 12:53:04
real	0m0.063s
user	0m0.017s
sys	0m0.028s
986 rmills@rmillsm1:~/Pictures/Photos/2023/Trips/Easter $ 

So you can read your 100,000 image library in about 100 seconds.

[GabZach]

Thanks a lot for the insights!

I was just wondering since I had done a test with ~10k images on a 2.5 inch external HD. The timings I got was 110 sec for calling exiv2 (no options) on 10373 image files. But I guess now that is mostly due to the slow bandwidth from the HD to my Macbook (M2 chip).

[clanmills]

@GabZach It's probably the latency of the disk. My MacBookPro has SSD storage which is fast and smooth, and nothing moves except bytes.

When I run the exact search on my USB Western Digital Drive attached to a ten-year-old MacMini:

505 rmills@rmillsmm-local:/Volumes/Backup/Photos/2023/Trips/Easter $ time find . -print0 -name "*.jpg" | xargs -0 exiv2 -g Exif.Photo.DateTimeOrig 2>/dev/null | wc
     670    4038   70939
real	0m9.559s
506 rmills@rmillsmm-local:/Volumes/Backup/Photos/2023/Trips/Easter $ 

One hundred (100) images/second on a portable drive. Please thank Andreas for the architecture that delivers such impressive performance.

Case closed?

[GabZach]

Yes, thanks a lot for your insights!

[clanmills]

I've had another two thoughts about this overnight.

1. Multithreading
   You could use multithreading. The code is thread-safe and re-entrant. There's an MT sample in the 0.27 code base. How many threads can you spin up? About 20 as it's restricted by <stdio.h> FILE objects. I suspect MT will give you impressive performance on your M2/SSD. I expect MT will buy little on your external drive.

2. Copy the metadata to a database
   I have 80,000 photos of my website https://clanmills.com. I have an SQLite database with a copy of a subset of the metadata. If I look for the 100 photos on my wedding anniversary (3/27) on the Easter Trip (we went to Iona in Scotland)

509 rmills@rmillsm1:~/gnu/github/exiv2/main $ time searchPhotos "2023:03:27"  | sort | head
sqlite3 -box '/Users/rmills/clanmills/photos.db' "SELECT * FROM photos WHERE title  LIKE '%2023:03:27%' OR taken LIKE '%2023:03:27%' OR path LIKE '%2023:03:27%' ORDER by taken DESC;"
│ /2023/Trips/Easter/Iona/dsc_3488.jpg     │ 2023:03:27 07:26:49 │ View at breakfast         │
│ /2023/Trips/Easter/Iona/dsc_3489.jpg     │ 2023:03:27 07:32:06 │ Mull at Breakfast         │
│ /2023/Trips/Easter/Iona/dsc_3490.jpg     │ 2023:03:27 08:49:26 │ Map of Oban               │
│ /2023/Trips/Easter/Iona/dsc_3491.jpg     │ 2023:03:27 08:56:54 │ Boarding Coruisk          │
│ /2023/Trips/Easter/Iona/dsc_3492.jpg     │ 2023:03:27 08:57:54 │ Cardeck                   │
│ /2023/Trips/Easter/Iona/dsc_3493.jpg     │ 2023:03:27 08:59:41 │ Photo of Coruisk          │
│ /2023/Trips/Easter/Iona/dsc_3494.jpg     │ 2023:03:27 09:16:22 │ McCaig's Tower            │
│ /2023/Trips/Easter/Iona/dsc_3495.jpg     │ 2023:03:27 09:23:59 │ Oban Harbour              │

real	0m0.437s
510 rmills@rmillsm1:~/gnu/github/exiv2/main $ 

I've only rebuilt the SQLite database once and it took about 8 hours for the python scripts to execute exiv2 about 80,000 times and update the various tables in the database. However, we're now discussing application design and not exiv2. Returning to your question "does Exiv2 read the whole file?", the answer is "No. It's optimized to minimize I/O".

[GabZach]

I've had another two thoughts about this overnight.

Thanks a lot for your thoughts!

• Multithreading I expect MT will buy little on your external drive.

I suspect so, too, and this is the important case, since the SSD is fast enough.

• Copy the metadata to a database

That is what I am doing. But customers have "complained" that scanning of 100k images takes about 10-20 minutes when doing that first scan to gather the database. After that, everything is smooth, of course.

of the metadata. If I look for the 100 photos on my wedding anniversary (3/27) on the Easter Trip (we went to Iona in Scotland)

great tour! When I was 19 I took a trip with my brother to Scotland, too; we never reached Iona, but went to Mull, Oban, Inverarray, etc. Very memorable trip! Best regards, Gabriel

[clanmills]

@GabZach People will complain about almost anything. If they can piss and moan, they do. On the other hand, I look at their efforts and see little evidence that their code/systems reach the perfection they demand from me. For example, a user bitched that the MT support could not handle thousands of threads. It's not only the limit of FILE objects in stdio.h, macOS has a limit to the number of open streams.

Anyway, on a happier note, I'm now retired from Exiv2 and looking forward to our Golden Wedding next April. The photo for the invitation was taken on the trip to Iona. Life is good. Don't let the moaners get you down.

[GabZach]

@GabZach People will complain about almost anything. If they can piss and moan, they do.

;-)

Anyway, on a happier note, I'm now retired from Exiv2 and looking forward to our Golden Wedding next April.

Congratulations! Quite an achievement, if you will. (Not many people, including myself, reach that.)

Life is good. Don't let the moaners get you down.—

Very good advice. Happy festivities. Best, G.