Archive format

[Munter]

We've talked on multiple occasions about archiving formats to create assetgraph snapshots, spare some server roundtrips or just keep around for later detailed analysis.

Requirements

• File transfer metadata must be preserved
  • HTTP request headers
  • HTTP response headers
  • File system stat information?
• Asset content must be available
  • Possibly decouple metadata from data itself to enable quick access
• Assetgraph asset properties should be saved for possible reuse without parsing the raw source again
• Asset relations should be available
  • We can't keep the relations node reference alive
  • from and to could probably be serialized to URI's

** WORK IN PROGRESS **

Canditates / Inspiration

https://w3c.github.io/web-performance/specs/HAR/Overview.html
https://www.w3.org/TR/web-packaging/
https://wicg.github.io/webpackage/draft-yasskin-dispatch-web-packaging.html

Possible features

• Transfer a graph from one place to another with all debugging information intact
• Fork a graph transformation pipeline to multiple processes or machines
• Trade memory usage for disk usage in high volume graphs
• Use an archive as a cache lookup when populating online resources
• Keep http conversation and file system metadata intact

Crazy ideas

• Merkle tree type implementation to incrementally add new differences while keeping old assets