You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The schema of a source is set on creation. As the upstream schema changes, Materialize either tolerates these changes or puts the source in an errored state, but doesn't explicitly flag that a newer schema version is available. To help identify (sub)sources that have newer schemas available, we should provide users with context on upstream schema versioning.
For Kafka sources using a schema registry, we can write down the schema version used when the source was created. Ideally, we'd also ping the registry for the latest version on some cadence, so we can flip on a bool to signal there is a newer version available.
For PostgreSQL and MySQL sources, we can flip on a bool when we receive a new Relation message in the replication stream.
Once this is available in the system catalog, we can use it to annotate (sub)sources in the Console, as a first step. cc @ggnall@parkerhendo
The text was updated successfully, but these errors were encountered:
I'm not entirely certain about the implementation here. It may be simpler/better/easier to periodically ping the upstream system to get the table's current schema and diff it against the schema in Materialize.
For PostgreSQL and MySQL, we might want to wait until sources and subsources get untangled on the statistics side before implementing this. Gus will get an estimate for the Kafka bit, which is the most pressing user need for the short-term.
Feature request
The schema of a source is set on creation. As the upstream schema changes, Materialize either tolerates these changes or puts the source in an errored state, but doesn't explicitly flag that a newer schema version is available. To help identify (sub)sources that have newer schemas available, we should provide users with context on upstream schema versioning.
version
used when the source was created. Ideally, we'd also ping the registry for the latestversion
on some cadence, so we can flip on a bool to signal there is a newer version available.Once this is available in the system catalog, we can use it to annotate (sub)sources in the Console, as a first step. cc @ggnall @parkerhendo
The text was updated successfully, but these errors were encountered: