Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add stateless reader domains #103

Open
jonhoo opened this issue Sep 19, 2018 · 0 comments
Open

Add stateless reader domains #103

jonhoo opened this issue Sep 19, 2018 · 0 comments
Labels
enhancement New feature or request m-materialization Related to materialization (full or otherwise) m-operators Related to operator implementations and semantics research This issue represents an open research question, and will require careful thought to fix.

Comments

@jonhoo
Copy link
Contributor

jonhoo commented Sep 19, 2018

It came up in a discussion today that it'd be neat to have stateless domains below Reader nodes that reads could "process" things they extract from Reader state through. Since the domain is stateless, readers could all process this data-flow in parallel using combinations of data they've read without worrying about synchronization with other executors. This would allow us to move certain compute to the readers, which might let us materialize a less verbose Reader further "up" the graph (e.g., above an expensive join).

@jonhoo jonhoo added enhancement New feature or request research This issue represents an open research question, and will require careful thought to fix. m-operators Related to operator implementations and semantics m-materialization Related to materialization (full or otherwise) labels Sep 19, 2018
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request m-materialization Related to materialization (full or otherwise) m-operators Related to operator implementations and semantics research This issue represents an open research question, and will require careful thought to fix.
Projects
None yet
Development

No branches or pull requests

1 participant