New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Stop OffsetStore when stopping the connector #12457
Conversation
@dlg99:Thanks for your contribution. For this PR, do we need to update docs? |
@dlg99:Thanks for providing doc info! |
if (null != producer) { | ||
try { | ||
producer.flush(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Can we put the flush and close in the same try-catch block?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
@hangc0276 in that case exception from flush prevent close(). We do want to close the producer.
it looks like a test is failing due to this patch
|
@eolivelli fixed. I did the patch on 2.7 branch originally, in master the client is passed through the context and should not be closed by the connector. |
### Motivation Source connectors based on KCA (all debezium ones) don't stop properly on error / don't restart. apache#12441 fixes one problem, this PR fixes another: ofsetStore is not closed on connector stop() and producer/consumer aren't closed too, preventing the connector from shutting down. ### Modifications Closing offset store on connector stop. (cherry picked from commit 63454e9)
Source connectors based on KCA (all debezium ones) don't stop properly on error / don't restart. apache#12441 fixes one problem, this PR fixes another: ofsetStore is not closed on connector stop() and producer/consumer aren't closed too, preventing the connector from shutting down. Closing offset store on connector stop. (cherry picked from commit 63454e9)
* up/master: (46 commits) [website][upgrade]feat: docs migration - version-2.7.2 Pulsar Schema (apache#12393) [docs] io-develop, fix broken link (apache#12414) docs(function): fix incorrect classname in python runtime sample (apache#12476) Remove redundant null check for getInternalListener (apache#12474) Fix the retry topic's `REAL_TOPIC` & `ORIGIN_MESSAGE_ID` property should not be modified once it has been written. (apache#12451) [cli] Fix output format of string by pulsar-admin command (apache#11878) fix the race of delete subscription and delete topic (apache#12240) fix influxdb yaml doc (apache#12460) [Modernizer] Add Maven Modernizer plugin in pulsar-proxy module (apache#12326) fix DefaultCryptoKeyReaderTest can not run on windows (apache#12475) apache#12429 only fixed the compactor skips data issue, but the normal reader/consumer (apache#12464) broker resource group test optimize fail msg (apache#12438) Stop OffsetStore when stopping the connector (apache#12457) fix a typo in UnAckedMessageTracker (apache#12467) docs(function): fix typo in pip install (apache#12468) Optimize the code: remove extra spaces (apache#12470) optimize SecurityUtility code flow (apache#12431) Update lombok to 1.18.22 (apache#12466) Update team.js to add David K. as a committer (apache#12440) Fix java demo error in reset cursor admin (apache#12454) ... # Conflicts: # site2/website-next/versioned_docs/version-2.7.2/schema-evolution-compatibility.md # site2/website-next/versioned_docs/version-2.7.2/schema-get-started.md # site2/website-next/versioned_docs/version-2.7.2/schema-manage.md # site2/website-next/versioned_docs/version-2.7.2/schema-understand.md # site2/website-next/versioned_sidebars/version-2.7.2-sidebars.json
Source connectors based on KCA (all debezium ones) don't stop properly on error / don't restart. #12441 fixes one problem, this PR fixes another: ofsetStore is not closed on connector stop() and producer/consumer aren't closed too, preventing the connector from shutting down. Closing offset store on connector stop. (cherry picked from commit 63454e9)
### Motivation Source connectors based on KCA (all debezium ones) don't stop properly on error / don't restart. #12441 fixes one problem, this PR fixes another: ofsetStore is not closed on connector stop() and producer/consumer aren't closed too, preventing the connector from shutting down. ### Modifications Closing offset store on connector stop. (cherry picked from commit 63454e9)
### Motivation Source connectors based on KCA (all debezium ones) don't stop properly on error / don't restart. apache#12441 fixes one problem, this PR fixes another: ofsetStore is not closed on connector stop() and producer/consumer aren't closed too, preventing the connector from shutting down. ### Modifications Closing offset store on connector stop.
Motivation
Source connectors based on KCA (all debezium ones) don't stop properly on error / don't restart.
#12441 fixes one problem, this PR fixes another: ofsetStore is not closed on connector stop() and producer/consumer aren't closed too, preventing the connector from shutting down.
Modifications
Closing offset store on connector stop.
Verifying this change
This change is a trivial rework / code cleanup without any test coverage.
Does this pull request potentially affect one of the following parts:
NO
If
yes
was chosen, please highlight the changesDocumentation
no-need-doc