Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Aggregate index support in Cascades #1864

Merged
merged 45 commits into from Nov 7, 2022

Conversation

hatyo
Copy link
Contributor

@hatyo hatyo commented Oct 12, 2022

This resolves #1885 and adds initial support to planning aggregate indexes (representing the GroupingKeyExpression key expression) to Cascades.

The support requires a query to have almost identical matching of the key expression.

Implementation notes:

  • Similar to RelationalExpression::subsumedBy, a subsumedBy method is added to Value indicating a subsumption relationship between Value instances, which could be useful as it provides more flexibility in calculating Value-compensation.
  • FieldValue employs a list of integers indicating the ordinal of each field in its field path, this list is used to check whether a FieldValue subsumes another FieldValue.
  • Lift various physical implementations of NumericAggregationValue such as sum and avg into their own internal classes so we can annotate them with either StreamableAggregateValue (a value that can be constructed using streaming aggregation) and / or IndexableAggregateValue (a value that can be backed by an aggregate index). This helps with pattern matching, constructing plans, and choosing the right match candidate. Similar work done to CountValue.
  • Added more Matchers for GroupBy constituents.
  • The generation of the Uber-record is adapted slightly to align it with the addition of field ordinals in FieldValue. It is just a temporary solution though until we have a proper fix, as explained in Uber-record type for FullUnorderedScanExpression is brittle #1884. Had to adapt a number of now-broken plan hash codes in tests because of this.
  • subsumption in GroupByExpression by a match candidate is, for now, exact. This means, for a match to happen, the query must match as-is the match candidate of the respective aggregate index. We will improve on this in the future and make it more robust.

@foundationdb-ci

This comment was marked as outdated.

@foundationdb-ci

This comment was marked as outdated.

@foundationdb-ci

This comment was marked as outdated.

@foundationdb-ci

This comment was marked as outdated.

@foundationdb-ci

This comment was marked as outdated.

@foundationdb-ci

This comment was marked as outdated.

@foundationdb-ci

This comment was marked as outdated.

@foundationdb-ci

This comment was marked as outdated.

@foundationdb-ci
Copy link
Contributor

Result of fdb-record-layer-pr-proto3 on Linux CentOS 7

  • Commit ID: 849b37a
  • Duration 0:07:20
  • Result: ❌ FAILED
  • Error: Error while executing command: ./gradlew --no-daemon --console=plain -b ./build.gradle build destructiveTest sonarqube -PcoreNotStrict -PreleaseBuild=false -PpublishBuild=false -PspotbugsEnableHtmlReport. Reason: exit status 1
  • Build Logs (available for 30 days)
  • Build Artifact (available for 30 days)

@foundationdb-ci
Copy link
Contributor

Result of fdb-record-layer-pr-proto2 on Linux CentOS 7

  • Commit ID: 849b37a
  • Duration 1:01:44
  • Result: ❌ FAILED
  • Error: Build has timed out.
  • Build Logs (available for 30 days)
  • Build Artifact (available for 30 days)

@foundationdb-ci
Copy link
Contributor

Result of fdb-record-layer-pr-proto2 on Linux CentOS 7

  • Commit ID: 4835cea
  • Duration 0:10:45
  • Result: ❌ FAILED
  • Error: Error while executing command: ./gradlew --no-daemon --console=plain -b ./build.gradle build destructiveTest sonarqube -PcoreNotStrict -PreleaseBuild=false -PpublishBuild=false -PspotbugsEnableHtmlReport. Reason: exit status 1
  • Build Logs (available for 30 days)
  • Build Artifact (available for 30 days)

@foundationdb-ci
Copy link
Contributor

Result of fdb-record-layer-pr-proto3 on Linux CentOS 7

  • Commit ID: 4835cea
  • Duration 0:11:30
  • Result: ❌ FAILED
  • Error: Error while executing command: ./gradlew --no-daemon --console=plain -b ./build.gradle build destructiveTest sonarqube -PcoreNotStrict -PreleaseBuild=false -PpublishBuild=false -PspotbugsEnableHtmlReport. Reason: exit status 1
  • Build Logs (available for 30 days)
  • Build Artifact (available for 30 days)

normen662
normen662 previously approved these changes Nov 7, 2022
Copy link
Contributor

@normen662 normen662 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great stuff!

@foundationdb-ci
Copy link
Contributor

Result of fdb-record-layer-pr-proto2 on Linux CentOS 7

@foundationdb-ci
Copy link
Contributor

Result of fdb-record-layer-pr-proto3 on Linux CentOS 7

@foundationdb-ci
Copy link
Contributor

Result of fdb-record-layer-pr-proto2 on Linux CentOS 7

@sonarcloud
Copy link

sonarcloud bot commented Nov 7, 2022

SonarCloud Quality Gate failed.    Quality Gate failed

Bug A 0 Bugs
Vulnerability A 0 Vulnerabilities
Security Hotspot A 0 Security Hotspots
Code Smell A 94 Code Smells

0.0% 0.0% Coverage
4.8% 4.8% Duplication

@foundationdb-ci
Copy link
Contributor

Result of fdb-record-layer-pr-proto3 on Linux CentOS 7

@normen662 normen662 merged commit 32accd4 into FoundationDB:main Nov 7, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Support planning aggregate indexes
3 participants