New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
feat: add table.createInsertStream for native streaming inserts #997
Conversation
maxOutstandingBytes: 1 * 1024 * 1024, | ||
|
||
// The maximum time we'll wait to send batched rows, in milliseconds. | ||
maxDelayMillis: 10000, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
How was 10 seconds chosen?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Just based off similar batching of messages in nodejs-pubsub, do you have an idea of a timeframe that is better aligned for BigQuery inserts?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not really. I think it has more to do with the customer's requirements than BigQuery limitations. Since it's configurable, aligning the default with pub/sub makes sense to me.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sorry for the flood of comments ^^; I'm happy to talk over any of them and/or help fix some of the any
comments if you like.
…into createInsertStream
…into createInsertStream
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I can't remember what all I commented on now, but overall it looks good :)
const opts = typeof options === 'object' ? options : {}; | ||
|
||
if (opts.insertRowsOptions) { | ||
this.insertRowsOptions = opts.insertRowsOptions; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Ah geez, I never actually gave this link to you... Here's one method:
}; | ||
}), | ||
// eslint-disable-next-line @typescript-eslint/no-explicit-any | ||
row: rows[(insertError as any).index], |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Sounds good :)
maxOutstandingBytes: 1 * 1024 * 1024, | ||
|
||
// The maximum time we'll wait to send batched rows, in milliseconds. | ||
maxDelayMillis: 10000, |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Not really. I think it has more to do with the customer's requirements than BigQuery limitations. Since it's configurable, aligning the default with pub/sub makes sense to me.
This feature adds the ability to use a native Duplex stream for inserting rows into BigQuery via the
/insertAll
endpoint and reading the API response. Implements batching of rows viaRowBatch
andRowQueue
classes.Adds:
Table.createInsertStream()
RowQueue
RowBatch
Ensure the tests and linter pass
Code coverage does not decrease (if any source code was changed)
Appropriate docs were updated (if necessary)
Fixes #506 🦕