You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Still need to think through whether fingerprints should be calculated from View (in-memory) or Storage (on-disk) data.
Considerations:
Portability: View ✅
View Fingerprints can be stable across Format/Storage changes (thus avoiding downstream invalidation)
If careful, fingerprinting should be stable across different Views
Flexibility: View ✅
Not all storage systems will support much more than mtime or their own custom checksum (eg: MD5 or CRC32C for GCS/HDFS, mtime only in BQ or local disks, etc)
Cost: Storage ✅
Storage based fingerprints will often just be a metadata lookup vs a full read for a View
Perhaps if we always track mtime+fingerprint, we can default to assuming mtime based immutability and only compute the real fingerprint on first write or mtime change (when something outside the system mutates data). For things with skewed mtime (ex: a dir of files), we'd probably track the latest one.
The text was updated successfully, but these errors were encountered:
Still need to think through whether fingerprints should be calculated from View (in-memory) or Storage (on-disk) data.
Considerations:
Perhaps if we always track mtime+fingerprint, we can default to assuming mtime based immutability and only compute the real fingerprint on first write or mtime change (when something outside the system mutates data). For things with skewed mtime (ex: a dir of files), we'd probably track the latest one.
The text was updated successfully, but these errors were encountered: