Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Records correlation in the after method #1637

Open
itiulkanov opened this issue Jul 27, 2022 · 0 comments
Open

Records correlation in the after method #1637

itiulkanov opened this issue Jul 27, 2022 · 0 comments

Comments

@itiulkanov
Copy link

Is it possible to correlate records that were sent using batching mode with the array items (req []elastic.BulkableRequest or response *elastic.BulkResponse) in the "after" method? E.g. attach some metadata from the record when sending to the index, and retrieve that metadata in the "after" method, to post-process only the particular failed records.

We are using
[ ] elastic.v7 (for Elasticsearch 7.x)

Please describe the expected behavior

We expect to receive some metadata we can use to post-process the failed records. For example, we had a dead node and we are willing to send data to DLQ. To do so we need to have some clues on which of the records from the particular batch were failing and correlate them with the records that were put in a batch

Please describe the actual behavior

At the first glance, there are no specific values I can use to do the correlation neither from elastic.BulkableRequest nor elastic.BulkResponse, and I can only stop the process, return an error or do some other operations not related to particular unindexed records.

Any steps to reproduce the behavior?

NA

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant