Enhancements to SchemaSync for iModel Hub Forking, Initialization, and Profile Upgrades

[khanaffan]

IModel Hub initializing & forking usecase

1. SyncId now mean ContainerId which of type string (instead of BeGuid) and must be provided by caller when initializing schema sync container. Previously it was random Guid that was problem when forking iModel from specific changeset.
2. For any given iModel, a schema sync container can be reinitialized with a new container. In this scenario, SchemaSync will allocate a new containerId to SyncId and will also transfer the DataVer from the iModel to the new container.

Profile upgrades

When an iModel undergoes a profile upgrade, it is now also pushed to the schema sync container, whereas previously it was not.

1. Under the new rule profile, versions of schema sync containers and iModels must adhere to the following rules.
   • On pull: The iModel ECDb profile version should be less than or equal to the schema sync container ECDb profile version.
   • On push: The iModel ECDb profile version should be greater than or equal to the schema sync container ECDb profile version.
2. The IModelDb.upgradeSchemas() function in iTwin ensures that the profile upgrade is distributed to the users.

Other important changes

1. When merging local transactions, we now include all DDLs, which was not the case previously for some reason. We only included first DDL txn and then ignore rest.
2. When applying a changeset, apply 'ec_' changes before data changes, and after applying 'ec_' changes, synchronize the database schema to ensure that subsequent data can be inserted correctly. This is necessary when the schema and data are in the same changeset.
3. In the past, only changesets with DDL were internally recognized as schema changesets when applying a changeset. However, changes to 'ec_*' tables without DDL were not considered as schema changesets. This meant that schema caches were sometimes not cleared when there were no DDL changes. Now, we use ChangesetType to ensure that both EC and DDL changes are treated as schema changes.

Changes

• If SchemaSync is enabled for iModel then all push schema changeset will have type ChangesetType.SchemaSync which is ChangesetType.Schema | 64 = 65.
• IModelDb.pullChanges() automatically does SchemaSync.pull() after incoming changeset are applied.
• IModelDb.pushChanges() automatically does SchemaSync.pull() before pushing changes.
• Added method SchemaSync.pull(). This allow application to pull new schema changes at any time.
• SchemaSync.setTestCache() now store cache dir in be_Local. This is done to allow testing of profile/schema upgrade which require a SchemaSync container lock while iModelDb connection is in close state.
• Changes to SchemaSync.initializeForIModel()
  • Require schema lock.
  • Take addition parameter ContainerId which is use as SyncId.
  • Will automatically abandon changes when error encounter. (previously file prop for container was set even when init itself failed)
  • overrideContainer flag allow SchemaSync.initializeForIModel() to override current SchemaSync container and initialize new one.
• Changes to changeset conflict handler
  • ec_* Data & Primary key conflict is skipped.
  • be_Prop conflict for (namespace: ec_Db & name: localDbInfo) is replaced.
• Changes IModelDb.doUpgrade().
  • When SchemaSync is enabled profile/domain schema upgrade happen while holding lock on SchemaSync container. This prevent two users from running profile upgrade together.
  • After profile/ domain schema upgrade changes are push to SchemaSync container.
  • At the end changeset if any will be pushed to hub.

imodel-native: iTwin/imodel-native#744

[mergify]

This pull request is now in conflicts. Could you fix it @khanaffan? 🙏
To fixup this pull request, you can check out it locally. See documentation: https://help.github.com/articles/checking-out-pull-requests-locally/

[aruniverse]

This integration test is failing on every PR, I'm going to bypass policies and merge this PR.

1) RealityDataAccess (#integration)
     should get RealityDataSource for reality data attachment in iModel:
   AssertionError: expected undefined not to be undefined
    at Context.<anonymous> (home/vsts/work/1/s/full-stack-tests/core/src/frontend/standalone/RealityDataAccess.test.ts:252:27)