adapter/compute/storage: cardinality statistics frequently timeout

[mgree]

What version of Materialize are you using?

v0.97

What is the issue?

The CachedStatsOracle often works locally, but times out even with substantially higher timeouts.

Stats are currently ignored in join implementation, so this is not high impact.

A few thoughts on what could be causing the problem here, from discussion with @bkirwi:

• timestamp selection (we use query_as_of, which may be too early) and/or interaction between strict serializability and timestamp selection
• actually slow communication around stats collection
• programmer error of async/timeouts

[bkirwi]

To expand on a couple of things:

• In general it's extremely possible for the query to select a timestamp that's not available in the sources yet, especially under strict serializability. That means the query will block -- which is fine, but we'd prefer the optimizer didn't.
  • OTOH, we probably don't want the stats at the latest available frontier, incase the sources are way ahead.
  • One possibility we discussed is using the query_as_of as an upper bound.
• It sounded like we were seeing timeouts even when the timeout was set at multiple seconds... which seems too high for the above to explain.