Add last written position as metadata in snapshot

[deepthidevaki]

Description

Ref:- #9906 (comment)

For backup manager to identify if a snapshot can be included in the snapshot, a snapshot must include the position of the followup event included in the snapshot as part of its metadata. This event is the event whose source position = lastProcessedPosition in the snapshot. Since it is not possible to find actual "lastProcessedPosition" in the snapshot without opening the database, we can approximate it by taking "lastWrittenPosition" used by the AsyncSnapshotDirector to commit the snapshot. This value might be an over approximation, but it is safe to use it.

[deepthidevaki]

Here is what I did to achieve this:

• Added a PersistedSnapshot::getMetadata which returns a SnapshotMetadata
• SnapshotMetadata is written to a file named ZEEBE.METADATA in the snapshot path.
• SnapshotMetadata is serialized as json. This way we can also add new fields later without breaking compatibility.
• The metadata file is inside the snapshot. So it is replicated as a normal "SnapshotChunk". This way the replication protocol remains unchanged => backward compatible. On the receiver side, it can recognize the metadata file from its name and build a SnapshotMetadata object by reading the file.
• For backward compatibility, we handle the cases where metadata file is not available by building an incomplete SnapshotMetadata with known values.
• SnapshotMetadata tentatively consists of following
  • processedPosition (same as in snapshot id)
  • exportedPosition (same as in snapshotid)
  • lastFollowupEventPosition (provided by AsyncSnapshotDirector)
  • version (we have a version in PersistedSnapshot, but it is not used and not persisted. By persisting it in metadata we can use it later for backwards compatibity)

Snapshot directory content with metadata will be as follows

> tree data/raft-partition/partitions/1/snapshots/
data/raft-partition/partitions/1/snapshots/
├── 19-1-39-19
│   ├── 000008.log
│   ├── 000009.sst
│   ├── CURRENT
│   ├── MANIFEST-000005
│   ├── OPTIONS-000007
│   └── ZEEBE.METADATA
└── 19-1-39-19.checksum

[npepinpe]

Looks sounds. The only point is the JSON serialization. It has nice properties (human readable, editable, etc.) but I'm not sure it solves any form of schema evolution. Sure, you can add fields, but you could do that with MsgPack, or Protobuf, or YAML, or even SBE.

As we aren't reading often from it, I think the benefits of human readability trumps performance, so I don't think SBE would be particularly beneficial here. But if we really want to tackle schema evolution, I don't think JSON alone is enough for that. If backwards/forwards compatibility is important, then we need to explicitly have some form of update test to guarantee it. Just my two cents.

[deepthidevaki]

Human readability was definitely the main motivation to chose json. Also I think handling backward/forward compatibility is easy as long as we are only adding new fields by setting ignore unknown properties = true. I think, we can also remove fields because it is possible to deserialize a partial json object. Agree that we need tests to ensure compatibility.