Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Cannot use database after moving from local machine to NAS #1507

Open
odedia opened this issue Apr 9, 2024 · 12 comments
Open

Cannot use database after moving from local machine to NAS #1507

odedia opened this issue Apr 9, 2024 · 12 comments
Labels
bug Something isn't working

Comments

@odedia
Copy link

odedia commented Apr 9, 2024

I'm used my local machine to do a first run at my library, and then migrated the entire folder to my NAS (TrueNAS, ZFS, shared via SMB).

I then tried to run the command again using --update --ramdb, but it seems like osxphotos couldn't access the database.
This is the error I get:

> osxphotos export /Volumes/Data/Pictures/PhotoPrism/Originals \
--directory "{created.year}/{created.year}-{created.mm}-{created.dd}" \
--retry 3 --sidecar XMP \
--keyword-template "{label}" \
--keyword-template "{keyword}" \
--keyword-template "{person}" \
--replace-keywords \
--sidecar-drop-ext --download-missing --use-photokit \
--edited-suffix “_edited” --update --cleanup \
--ramdb
Using last opened Photos library: /Volumes/ExternalData/Users/odedia/Pictures/Photo Library.photoslibrary
Something went wrong and osxphotos encountered an error:
RetryError[<Future at 0x10b2b4ad0 state=finished raised OperationalError>]
Crash log written to '/Users/odedia/osxphotos_crash.log'
Please file a bug report at https://github.com/RhetTbull/osxphotos/issues with the crash log attached.

This is the contents of osxphotos_crash.log:

> cat /Users/odedia/osxphotos_crash.log
osxphotos crash log
Created: 2024-04-09 11:32:57.278663
osxphotos version: 0.67.10
Platform: macOS-14.4.1-x86_64-i386-64bit
Python version: 3.12.1 (v3.12.1:2305ca5144, Dec  7 2023, 17:23:39) [Clang 13.0.0 (clang-1300.0.29.30)]
sys.argv: ['osxphotos', 'export', '/Volumes/Data/Pictures/PhotoPrism/Originals', '--directory', '{created.year}/{created.year}-{created.mm}-{created.dd}', '--retry', '3', '--sidecar', 'XMP', '--keyword-template', '{label}', '--keyword-template', '{keyword}', '--keyword-template', '{person}', '--replace-keywords', '--sidecar-drop-ext', '--download-missing', '--use-photokit', '--edited-suffix', '“_edited”', '--update', '--cleanup', '--ramdb']
CRASH_DATA:
locals: {'dest': '/Volumes/Data/Pictures/PhotoPrism/Originals', 'db': None, 'add_exported_to_album': None, 'add_missing_to_album': None, 'add_skipped_to_album': None, 'added_after': None, 'added_before': None, 'added_in_last': None, 'album': (), 'album_keyword': False, 'alt_db': None, 'alt_copy': False, 'append': False, 'fix_orientation': False, 'beta': False, 'burst': False, 'checkpoint': None, 'cleanup': True, 'cloudasset': False, 'config_only': False, 'convert_to_jpeg': False, 'crash_after': None, 'current_name': False, 'deleted': False, 'deleted_only': False, 'description': (), 'description_template': None, 'directory': '{created.year}/{created.year}-{created.mm}-{created.dd}', 'download_missing': True, 'dry_run': False, 'duplicate': False, 'edited': False, 'edited_suffix': '“_edited”', 'exif': (), 'exiftool': False, 'exiftool_merge_keywords': False, 'exiftool_merge_persons': False, 'exiftool_option': (), 'exiftool_path': None, 'export_as_hardlink': False, 'export_by_date': False, 'exportdb': None, 'external_edit': False, 'favorite': False, 'favorite_rating': False, 'filename_template': None, 'finder_tag_keywords': False, 'finder_tag_template': (), 'folder': (), 'force_update': False, 'from_date': None, 'from_time': None, 'has_comment': False, 'has_likes': False, 'has_raw': False, 'hdr': False, 'hidden': False, 'ignore_case': False, 'ignore_date_modified': False, 'ignore_exportdb': False, 'ignore_signature': False, 'in_album': False, 'incloud': False, 'is_reference': False, 'jpeg_ext': None, 'jpeg_quality': None, 'keep': (), 'keyword': (), 'keyword_template': ('{label}', '{keyword}', '{person}'), 'label': (), 'limit': None, 'live': False, 'load_config': None, 'location': False, 'max_size': None, 'min_size': None, 'missing': False, 'name': (), 'no_comment': False, 'no_description': False, 'no_exportdb': False, 'no_keyword': False, 'no_likes': False, 'no_location': False, 'no_place': False, 'no_progress': False, 'no_title': False, 'not_burst': False, 'not_cloudasset': False, 'not_edited': False, 'not_favorite': False, 'not_hdr': False, 'not_hidden': False, 'not_in_album': False, 'not_incloud': False, 'not_live': False, 'not_missing': False, 'not_panorama': False, 'not_portrait': False, 'not_reference': False, 'not_screenshot': False, 'not_selfie': False, 'not_shared': False, 'not_slow_mo': False, 'not_time_lapse': False, 'only_movies': False, 'only_new': False, 'only_photos': False, 'original_suffix': None, 'overwrite': False, 'panorama': False, 'person': (), 'person_keyword': False, 'place': (), 'portrait': False, 'post_command': (), 'post_command_error': None, 'post_function': (), 'preview': False, 'preview_if_missing': False, 'preview_suffix': None, 'print_template': (), 'query_eval': (), 'query_function': (), 'ramdb': True, 'regex': (), 'replace_keywords': True, 'report': None, 'retry': 3, 'save_config': None, 'screenshot': False, 'selfie': False, 'shared': False, 'export_aae': False, 'sidecar': ('xmp',), 'sidecar_drop_ext': True, 'sidecar_template': (), 'skip_bursts': False, 'skip_edited': False, 'skip_live': False, 'skip_original_if_edited': False, 'skip_raw': False, 'skip_uuid': (), 'skip_uuid_from_file': None, 'slow_mo': False, 'strip': False, 'theme': None, 'time_lapse': False, 'timestamp': False, 'title': (), 'tmpdir': None, 'to_date': None, 'to_time': None, 'touch_file': False, 'update': True, 'update_errors': False, 'use_photokit': True, 'use_photos_export': False, 'uti': None, 'uuid': (), 'uuid_from_file': None, 'verbose_flag': 0, 'xattr_template': (), 'year': (), 'syndicated': False, 'not_syndicated': False, 'saved_to_library': False, 'not_saved_to_library': False, 'shared_moment': False, 'not_shared_moment': False, 'shared_library': False, 'not_shared_library': False, 'selected': False, 'ctx': <click.core.Context object at 0x103d3fd40>, 'cli_obj': <osxphotos.cli.cli.CLI_Obj object at 0x10b231a90>}
osxphotos version: 0.67.10
Error: RetryError[<Future at 0x10b2b4ad0 state=finished raised OperationalError>]
Traceback (most recent call last):
  File "tenacity/__init__.py", line 382, in __call__
  File "osxphotos/export_db.py", line 1154, in _open_export_db
sqlite3.OperationalError: unable to open database file

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "osxphotos/crash_reporter.py", line 79, in wrapped
  File "osxphotos/cli/export.py", line 1115, in export
  File "osxphotos/cli/export.py", line 1790, in export_cli
  File "osxphotos/export_db.py", line 1085, in __init__
  File "tenacity/__init__.py", line 289, in wrapped_f
  File "tenacity/__init__.py", line 379, in __call__
  File "tenacity/__init__.py", line 326, in iter
tenacity.RetryError: RetryError[<Future at 0x10b2b4ad0 state=finished raised OperationalError>]

I then decided to copy the database file to my local machine and that seems to have worked:

> osxphotos export /Volumes/Data/Pictures/PhotoPrism/Originals \
--directory "{created.year}/{created.year}-{created.mm}-{created.dd}" \
--retry 3 --sidecar XMP \
--keyword-template "{label}" \
--keyword-template "{keyword}" \
--keyword-template "{person}" \
--replace-keywords \
--sidecar-drop-ext --download-missing --use-photokit \
--edited-suffix “_edited” --update --cleanup \
--exportdb ~/.osxphotos_export.db
Using last opened Photos library: /Volumes/ExternalData/Users/odedia/Pictures/Photo Library.photoslibrary
Warning: export database is '/Users/odedia/.osxphotos_export.db' but found '.osxphotos_export.db' in
/Volumes/Data/Pictures/PhotoPrism/Originals; using '/Users/odedia/.osxphotos_export.db'
Warning: export database '/Users/odedia/.osxphotos_export.db' is in a different directory than export destination
'/Volumes/Data/Pictures/PhotoPrism/Originals'
Using osxphotos export database: version 9.1 located at /Users/odedia/.osxphotos_export.db
Processing database /Volumes/ExternalData/Users/odedia/Pictures/Photo Library.photoslibrary/database/photos.db
Processing database /Volumes/ExternalData/Users/odedia/Pictures/Photo Library.photoslibrary/database/Photos.sqlite
Photos database version: 6000, 9.
Processing persons in photos.
Processing detected faces in photos.
Processing albums.
Processing keywords.
Processing photo details.
Processing import sessions.
Processing additional photo details.
Processing face details.
Processing photo labels.
Processing EXIF details.
Processing computed aesthetic scores.
Processing comments and likes for shared photos.
Processing moments.
Processing syndication info.
Processing shared iCloud library info
Done processing details from Photos library.
Exporting 78424 photos to /Volumes/Data/Pictures/PhotoPrism/Originals...
Exporting 78424 photos ╸━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━   2% 1:33:41

I still prefer to keep the database close to the photos so using the ramdisk option would be preferred. If there's anything I can do to correct this I'd be happy to try it out.
Thanks.

@RhetTbull
Copy link
Owner

Interesting, you're the second user in the last few days to report this. I cannot think of any reason this shouldn't work. The entire reason the export database is kept with the export is so the entire export tree can be easily moved while preserving export state and I've tested this. I will do some testing but don't have a NAS to test with so will see what i can replicate. The error OperationalError is actually coming from SQLite which OSXPhotos uses to store the export database.

@RhetTbull
Copy link
Owner

@odedia @cukal I have replicated this issue. I don't have a NAS but created an SMB share on another Mac on my network and followed these steps:

  1. Export to the local Mac with osxphotos export --update --library test.photoslibrary /local/path
  2. Use the command rsync -av --progress /local/path /Volumes/remote/path
  3. Ran osxphotos export --update --library test.photoslibrary /Volumes/remote/path

OSXPhotos crashed with OperationalError during sqlite3.Connection.execute trying to read the export databse.

I used the same rsync command to copy the export directory to a new path on the local Mac and the export worked as expected:

  1. rsync -av --progress /local/path /local/path2
  2. osxphotos export --update --library test.photoslibrary /local/path2

Then I created a new directory on the network volume and ran OSXPhotos against this:

  1. osxphotos export --update --library test.photoslibrary /Volumes/remote/path2

and this worked as expected. OSXPhotos created a new export database and did the export. Re-running the export with --update causes OSXPhotos to work as expected.

Finally, I used rsync to copy the export from step 6 (newly created export on remote volume) to the local mac and re-ran the export:

  1. osxphotos export --update --library test.photoslibrary /new/local/path

And this worked fine! So the issue appears to happen only when copying a SQLite database from the local machine to the remote machine.

Give this only happens when opening the copied SQLite database on the network volume I suspect this is some sort of weird SQLite thing but I will instrument the OSXPhotos code and see if I can get any clues.

@RhetTbull
Copy link
Owner

More data: I could not create this with a simple test:

  1. sqlite3 test.db "CREATE TABLE mytable (id INTEGER PRIMARY KEY, name TEXT, age INTEGER);"
  2. sqlite3 test.db "INSERT INTO mytable (id, name, age) VALUES (1, 'John', 30);"
  3. sqlite3 test.db ".dump": worked as expected
  4. run the code below: python3 sqltest.py test.db: worked as expected
  5. rsync -v test.db /Volumes/remote/test
  6. sqlite3 /Volumes/remote/test/test.db ".dump": worked as expected
  7. run python3 sqltest.py /Volumes/remote/test/test.db worked as expected.

sqltest.py:

import sqlite3
import sys

# Connect to SQLite database
try:
    db_name = sys.argv[1]
except IndexError:
    sys.exit('No database name provided')

conn = sqlite3.connect(db_name)

# Create a cursor object
cur = conn.cursor()

# Execute a SQL query to fetch all rows from mytable
cur.execute('SELECT * FROM mytable')

# Fetch all rows as a list of tuples
rows = cur.fetchall()

# Print all rows
for row in rows:
    print(row)

# Close connection
conn.close()

@odedia
Copy link
Author

odedia commented Apr 9, 2024

Interesting. I wonder if SQLite uses some file sockets that only work when it's not on a network drive.

@RhetTbull
Copy link
Owner

More debug output leads to more questions. The OSXPhotos export_db.py code opens the database and returns a Connection object but trying any query on the connection results in OperationalError. I'm stumped -- will have to do some more digging on this.

osxphotos --breakpoint osxphotos.export_db::ExportDB._open_export_db export --verbose --update --library ~/Pictures/Test-13.0.0.photoslibrary /Volumes/remote/test
Breakpoint added for osxphotos.export_db::ExportDB._open_export_db
osxphotos version: 0.67.10
Python version: 3.11.5 (main, Aug 25 2023, 01:57:11) [Clang 14.0.3 (clang-1403.0.22.14.1)]
Platform: macOS-13.5.1-x86_64-i386-64bit, 13.5.1
Verbose level: 1
> /Users/rhet/.local/pipx/venvs/osxphotos/lib/python3.11/site-packages/osxphotos/debug.py(61)debug_breakpoint()
-> return wrapped(*args, **kwargs)
(Pdb) s
--Call--
> /Users/rhet/.local/pipx/venvs/osxphotos/lib/python3.11/site-packages/osxphotos/export_db.py(511)_open_export_db()
-> def _open_export_db(
(Pdb) s
> /Users/rhet/.local/pipx/venvs/osxphotos/lib/python3.11/site-packages/osxphotos/export_db.py(520)_open_export_db()
(Pdb) n
> /Users/rhet/.local/pipx/venvs/osxphotos/lib/python3.11/site-packages/osxphotos/export_db.py(527)_open_export_db()
-> conn = self._get_db_connection(dbfile)
(Pdb) n
> /Users/rhet/.local/pipx/venvs/osxphotos/lib/python3.11/site-packages/osxphotos/export_db.py(528)_open_export_db()
-> self.was_created = False
(Pdb) conn
<sqlite3.Connection object at 0x114a62b60>
(Pdb) conn.execute("SELECT * FROM runs").fetchall()
*** sqlite3.OperationalError: unable to open database file
(Pdb)

@RhetTbull
Copy link
Owner

Interesting. I wonder if SQLite uses some file sockets that only work when it's not on a network drive.

Except that is does work when SQLite creates the database on the network drive. Very strange!

@RhetTbull RhetTbull added the bug Something isn't working label Apr 9, 2024
@RhetTbull
Copy link
Owner

The sqlite docs recommend against using SQLite over a network connection. That's why OSXPhotos implements --ramdb. Ultimately, I think OSXPhotos needs a way to detect a network volume and use --ramdb or copy the database to the local drive before running the export.

In the meantime, I recommend you use the --export-db option to store the export database locally while I debug this issue further.,

@cukal
Copy link

cukal commented Apr 9, 2024

Thanks for looking into it, glad I'm not alone in being stumped. I'll use the --export-db option with the actual photos stored on the network drive, visionary foresight for implementing that feature!

@odedia
Copy link
Author

odedia commented Apr 9, 2024

The thing is - in my original issue, I did provide the --ramdb parameter but it still failed. So seems like the implementaton of --ramdb needs to first copy the database to some local folder and only then load into memory.

@RhetTbull
Copy link
Owner

Yes, --ramdb relies on the SQLite backup command to create an in memory version which by definition must be able to read the database.

I think I need to just update instructions to not store export db on network volume and use --export-db

@RhetTbull
Copy link
Owner

See also #1419

Will need to test whether a DB copied to the NAS by user then copied back to the Mac by OSXPhotos can be opened properly.

@RhetTbull
Copy link
Owner

See also this

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants