New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: Clickhouse Storage Driver #1342
base: main
Are you sure you want to change the base?
feat: Clickhouse Storage Driver #1342
Conversation
WalkthroughThis update introduces a new ClickHouse storage driver for Go applications, enhancing database interaction capabilities. It includes comprehensive configuration options, methods for CRUD operations, and robust testing via GitHub Actions. The changes aim to streamline data handling and improve testing efficiency for Clickhouse and Cloudflare KV projects. Changes
Thank you for using CodeRabbit. We offer it for free to the OSS community and would appreciate your support in helping us grow. If you find it useful, would you consider giving us a shout-out on your favorite social media? TipsChatThere are 3 ways to chat with CodeRabbit:
Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (invoked as PR comments)
Additionally, you can add CodeRabbit Configration File (
|
Taking a look at the failing checks already |
@luk3skyw4lker The go.mod files are still broken. You edited the root one |
@gaby fixed it, sorry to take so long! |
@luk3skyw4lker The benchmark yml has to be updated to run clickhouse |
@gaby I added the clickhouse startup command, should be ok right now |
@gaby the latest commit should fix the error in the benchmark, I changed the clickhouse port to 9001 |
|
||
const createTableString = ` | ||
CREATE TABLE IF NOT EXISTS {table:Identifier} ( | ||
key String |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We should make this a function that allows specifying compression type/level. Clickhouse is known for allowing using zstd/lz compression
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'll check out some ways to do that. I haven't come across a create table query specifying the compression method but I'll research for it
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
About this, seems like the only engine where you could change the compression methods is the MergeTree family and seems a little bit overcomplicated to have all those treatments here. If you guys really want to support this it's possible but it's harder than it seemed initially: https://clickhouse.com/docs/en/sql-reference/statements/create/table#column-compression-codecs
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@gaby thoughts?
Few comments, benchmarks are working now 💪 |
I'm almost done with all the comments, probably will send some code by end of the week. Sorry to not do it sooner! |
@luk3skyw4lker Thanks! |
@luk3skyw4lker thanks, just ping us when you are done |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 6
Review Details
Configuration used: CodeRabbit UI
Review profile: CHILL
Files selected for processing (2)
- clickhouse/clickhouse.go (1 hunks)
- clickhouse/clickhouse_test.go (1 hunks)
Files not reviewed due to errors (1)
- clickhouse/clickhouse_test.go (no review received)
Additional comments not posted (3)
clickhouse/clickhouse.go (3)
103-110
: Ensure proper error handling in theDelete
method.The implementation of the
Delete
method is straightforward and correctly uses the context and parameters. Good job on keeping it concise and clear.
112-114
: Ensure proper error handling in theReset
method.The implementation of the
Reset
method is straightforward and correctly uses the context. Good job on keeping it concise and clear.
116-118
: Ensure proper error handling in theClose
method.The implementation of the
Close
method is straightforward and correctly handles the session closure. Good job on keeping it concise and clear.
if configuration.Clean { | ||
queryWithEngine := fmt.Sprintf(createTableString, engine) | ||
if err := conn.Exec(ctx, queryWithEngine); err != nil { | ||
return &Storage{}, err | ||
} | ||
|
||
if err := conn.Exec(ctx, resetDataString); err != nil { | ||
return &Storage{}, err | ||
} | ||
} |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Consider handling potential SQL injection vulnerabilities.
+ // Ensure that `engine` is sanitized or parameterized to prevent SQL injection.
+ // Example: Validate `engine` against a list of allowed values.
The engine
variable is directly used in a SQL query. This could lead to SQL injection if not properly sanitized or validated.
Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation.
if configuration.Clean { | |
queryWithEngine := fmt.Sprintf(createTableString, engine) | |
if err := conn.Exec(ctx, queryWithEngine); err != nil { | |
return &Storage{}, err | |
} | |
if err := conn.Exec(ctx, resetDataString); err != nil { | |
return &Storage{}, err | |
} | |
} | |
if configuration.Clean { | |
// Ensure that `engine` is sanitized or parameterized to prevent SQL injection. | |
// Example: Validate `engine` against a list of allowed values. | |
queryWithEngine := fmt.Sprintf(createTableString, engine) | |
if err := conn.Exec(ctx, queryWithEngine); err != nil { | |
return &Storage{}, err | |
} | |
if err := conn.Exec(ctx, resetDataString); err != nil { | |
return &Storage{}, err | |
} | |
} |
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 0
Review Details
Configuration used: CodeRabbit UI
Review profile: CHILL
Files selected for processing (1)
- clickhouse/clickhouse.go (1 hunks)
Files skipped from review as they are similar to previous changes (1)
- clickhouse/clickhouse.go
Co-authored-by: coderabbitai[bot] <136622811+coderabbitai[bot]@users.noreply.github.com>
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Actionable comments posted: 0
Review Details
Configuration used: CodeRabbit UI
Review profile: CHILL
Files selected for processing (1)
- clickhouse/clickhouse.go (1 hunks)
Files skipped from review as they are similar to previous changes (1)
- clickhouse/clickhouse.go
Adding clickhouse implementation, attending #1204 request
Summary by CodeRabbit
New Features
Tests
Documentation
Chores