Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

#683 Improve concurrency #1258

Merged
merged 82 commits into from
Mar 15, 2022
Merged

#683 Improve concurrency #1258

merged 82 commits into from
Mar 15, 2022

Commits on Nov 23, 2021

  1. refactor: remove currency concurrency and locking. clear restart time… 

    …out on consumer stop(). fix some memory leaks in tests
    @priitkaard
    priitkaard committed Nov 23, 2021
    Configuration menu
    Copy the full SHA
    3b74924 View commit details
    Browse the repository at this point in the history

  2. refactor: start runner via runnerPool. remove partitionsConsumedConcu… 

    …rrently usages from runner
    @priitkaard
    priitkaard committed Nov 23, 2021
    Configuration menu
    Copy the full SHA
    3141922 View commit details
    Browse the repository at this point in the history

Commits on Nov 24, 2021

  1. feat: experiment with concurrencyManager distribution algorithm

    @priitkaard
    priitkaard committed Nov 24, 2021
    Configuration menu
    Copy the full SHA
    4a3a00d View commit details
    Browse the repository at this point in the history

Commits on Dec 6, 2021

  1. feat: separate worker for concurrent fetching per broker

    @priitkaard
    priitkaard committed Dec 6, 2021
    Configuration menu
    Copy the full SHA
    4c4055e View commit details
    Browse the repository at this point in the history

Commits on Dec 7, 2021

  1. refactor: fetch worker for each broker

    @priitkaard
    priitkaard committed Dec 7, 2021
    Configuration menu
    Copy the full SHA
    b019bc3 View commit details
    Browse the repository at this point in the history

  2. refactor: reduce the amount of changes done to tests. Rename fetch to… 

    … consume in tests
    @priitkaard
    priitkaard committed Dec 7, 2021
    Configuration menu
    Copy the full SHA
    7b3a219 View commit details
    Browse the repository at this point in the history

  3. refactor: rename scheduleFetch to scheduleConsume

    @priitkaard
    priitkaard committed Dec 7, 2021
    Configuration menu
    Copy the full SHA
    61c0859 View commit details
    Browse the repository at this point in the history

  4. Merge remote-tracking branch 'origin/master' into concurrency

    @priitkaard
    priitkaard committed Dec 7, 2021
    Configuration menu
    Copy the full SHA
    544722c View commit details
    Browse the repository at this point in the history

  5. fix: remove interval hack

    @priitkaard
    priitkaard committed Dec 7, 2021
    Configuration menu
    Copy the full SHA
    6332d8d View commit details
    Browse the repository at this point in the history

  6. fix: mark offset out of range error retriable. merge fetch error hand… 

    …ling. cleanup fetcher queue on stop
    @priitkaard
    priitkaard committed Dec 7, 2021
    Configuration menu
    Copy the full SHA
    efdbaf6 View commit details
    Browse the repository at this point in the history

  7. refactor: improve concurrent crash handling

    @priitkaard
    priitkaard committed Dec 7, 2021
    Configuration menu
    Copy the full SHA
    11030e1 View commit details
    Browse the repository at this point in the history

Commits on Dec 8, 2021

  1. refactor: combine fetch queue. rename pool to fetch manager. revert o… 

    …ffset out of range to non-retriable
    @priitkaard
    priitkaard committed Dec 8, 2021
    Configuration menu
    Copy the full SHA
    64ddb41 View commit details
    Browse the repository at this point in the history

  2. refactor: restore more error handling

    @priitkaard
    priitkaard committed Dec 8, 2021
    Configuration menu
    Copy the full SHA
    e39603a View commit details
    Browse the repository at this point in the history

  3. test: uncomment some failing tests

    @priitkaard
    priitkaard committed Dec 8, 2021
    Configuration menu
    Copy the full SHA
    8907437 View commit details
    Browse the repository at this point in the history

  4. test: fix runner tests

    @priitkaard
    priitkaard committed Dec 8, 2021
    Configuration menu
    Copy the full SHA
    ecbd862 View commit details
    Browse the repository at this point in the history

  5. fix: safely leave consumer group on restart

    @priitkaard
    priitkaard committed Dec 8, 2021
    Configuration menu
    Copy the full SHA
    88d4b40 View commit details
    Browse the repository at this point in the history

  6. fix: avoid restart after disconnected by afterEach hooks

    @priitkaard
    priitkaard committed Dec 8, 2021
    Configuration menu
    Copy the full SHA
    7a1ca90 View commit details
    Browse the repository at this point in the history

  7. refactor: cleanup onCrash

    @priitkaard
    priitkaard committed Dec 8, 2021
    Configuration menu
    Copy the full SHA
    1e31962 View commit details
    Browse the repository at this point in the history

  8. test: fix rebalance without retries

    @priitkaard
    priitkaard committed Dec 8, 2021
    Configuration menu
    Copy the full SHA
    ef5a135 View commit details
    Browse the repository at this point in the history

  9. feat: distribute partitions across concurrent runners

    @priitkaard
    priitkaard committed Dec 8, 2021
    Configuration menu
    Copy the full SHA
    ec11fc8 View commit details
    Browse the repository at this point in the history

Commits on Dec 16, 2021

  1. refactor: change seekOffset data structure for easier separation betw… 

    …een brokers. move fetch instrumentation emitting to fetchManager. seek all offsets, even if paused
    @priitkaard
    priitkaard committed Dec 16, 2021
    Configuration menu
    Copy the full SHA
    731fbbc View commit details
    Browse the repository at this point in the history

  2. refactor: fetch per node + concurrent runner. 

    test issues resolved.
cleanup TODO.
    @priitkaard
    priitkaard committed Dec 16, 2021
    Configuration menu
    Copy the full SHA
    f0c9c96 View commit details
    Browse the repository at this point in the history

  3. refactor: cleanup hardcoded sleeps in runner tests

    @priitkaard
    priitkaard committed Dec 16, 2021
    Configuration menu
    Copy the full SHA
    45f81bf View commit details
    Browse the repository at this point in the history

  4. refactor: cleanup active/node partition filtering

    @priitkaard
    priitkaard committed Dec 16, 2021
    Configuration menu
    Copy the full SHA
    def6c95 View commit details
    Browse the repository at this point in the history

  5. refactor: cleanup

    @priitkaard
    priitkaard committed Dec 16, 2021
    Configuration menu
    Copy the full SHA
    b277ba6 View commit details
    Browse the repository at this point in the history

  6. Merge remote-tracking branch 'origin/master' into concurrency

    @priitkaard
    priitkaard committed Dec 16, 2021
    Configuration menu
    Copy the full SHA
    07450f7 View commit details
    Browse the repository at this point in the history

Commits on Dec 17, 2021

  1. refactor: minor changes before refactoring to single requests per bro… 

    …ker due to blocking fetch request limitations
    @priitkaard
    priitkaard committed Dec 17, 2021
    Configuration menu
    Copy the full SHA
    6872058 View commit details
    Browse the repository at this point in the history

  2. feat: socket for polls 

    separate socket for fetch polls to avoid blocking other concurrent requests
fetch all topicPartitions per broker together
track and don't fetch nodes that are being processed
    @priitkaard
    priitkaard committed Dec 17, 2021
    Configuration menu
    Copy the full SHA
    b783a62 View commit details
    Browse the repository at this point in the history

  3. 1.15.1

    @priitkaard
    priitkaard committed Dec 17, 2021
    Configuration menu
    Copy the full SHA
    1c87136 View commit details
    Browse the repository at this point in the history

Commits on Dec 20, 2021

  1. tulios#683 refactor: final renaming before PR

    @priitkaard
    priitkaard committed Dec 20, 2021
    Configuration menu
    Copy the full SHA
    5031975 View commit details
    Browse the repository at this point in the history

  2. tulios#683 test: fetch manager

    @priitkaard
    priitkaard committed Dec 20, 2021
    Configuration menu
    Copy the full SHA
    bf4c827 View commit details
    Browse the repository at this point in the history

  3. test: fix mock reset

    @priitkaard
    priitkaard committed Dec 20, 2021
    Configuration menu
    Copy the full SHA
    96bf844 View commit details
    Browse the repository at this point in the history

  4. Merge pull request #1 from priitkaard/concurrency 

    tulios#683 Improve concurrency
    @priitkaard
    priitkaard committed Dec 20, 2021
    Configuration menu
    Copy the full SHA
    d2128cc View commit details
    Browse the repository at this point in the history

  5. Merge branch 'tulios:master' into master

    @priitkaard
    priitkaard committed Dec 20, 2021
    Configuration menu
    Copy the full SHA
    cc9559c View commit details
    Browse the repository at this point in the history

  6. fix: flatMap implementation

    @priitkaard
    priitkaard committed Dec 20, 2021
    Configuration menu
    Copy the full SHA
    2c37d39 View commit details
    Browse the repository at this point in the history

  7. fix: improve error handling during concurrent rebalance

    @priitkaard
    priitkaard committed Dec 20, 2021
    Configuration menu
    Copy the full SHA
    eacb34a View commit details
    Browse the repository at this point in the history

  8. 1.16.1

    @priitkaard
    priitkaard committed Dec 20, 2021
    Configuration menu
    Copy the full SHA
    ef75150 View commit details
    Browse the repository at this point in the history

Commits on Dec 22, 2021

  1. refactor: clear onCrash timeout on stop()

    @priitkaard
    priitkaard committed Dec 22, 2021
    Configuration menu
    Copy the full SHA
    e622ed7 View commit details
    Browse the repository at this point in the history

Commits on Dec 28, 2021

  1. refactor: connection pool 

    * revert connection changes
* replace socket map with connection pool layer
* authenticate both sockets
    @priitkaard
    priitkaard committed Dec 28, 2021
    Configuration menu
    Copy the full SHA
    69fef0e View commit details
    Browse the repository at this point in the history

  2. test: spy a single connection

    @priitkaard
    priitkaard committed Dec 28, 2021
    Configuration menu
    Copy the full SHA
    0619844 View commit details
    Browse the repository at this point in the history

Commits on Dec 31, 2021

  1. fix: move default connectionTimeout to upper scope

    @priitkaard
    priitkaard committed Dec 31, 2021
    Configuration menu
    Copy the full SHA
    9bc80d3 View commit details
    Browse the repository at this point in the history

Commits on Jan 13, 2022

  1. Merge remote-tracking branch 'tulios/master'

    @priitkaard
    priitkaard committed Jan 13, 2022
    Configuration menu
    Copy the full SHA
    3aeb591 View commit details
    Browse the repository at this point in the history

Commits on Feb 9, 2022

  1. Merge remote-tracking branch 'tulios/master'

    @priitkaard
    priitkaard committed Feb 9, 2022
    Configuration menu
    Copy the full SHA
    4dbd48a View commit details
    Browse the repository at this point in the history

  2. Revert "Merge remote-tracking branch 'tulios/master'" 

    This reverts commit 4dbd48a, reversing
changes made to 3aeb591.
    @priitkaard
    priitkaard committed Feb 9, 2022
    Configuration menu
    Copy the full SHA
    69077ec View commit details
    Browse the repository at this point in the history

  3. revert last two commits

    @priitkaard
    priitkaard committed Feb 9, 2022
    Configuration menu
    Copy the full SHA
    eecee03 View commit details
    Browse the repository at this point in the history

  4. Merge branch 'master' of github.com:priitkaard/kafkajs

    @priitkaard
    priitkaard committed Feb 9, 2022
    Configuration menu
    Copy the full SHA
    1436605 View commit details
    Browse the repository at this point in the history

Commits on Feb 10, 2022

  1. merge tulios/master

    @priitkaard
    priitkaard committed Feb 10, 2022
    Configuration menu
    Copy the full SHA
    3b44fd2 View commit details
    Browse the repository at this point in the history

  2. commit stashed

    @priitkaard
    priitkaard committed Feb 10, 2022
    Configuration menu
    Copy the full SHA
    e9c39f6 View commit details
    Browse the repository at this point in the history

  3. Merge remote-tracking branch 'tulios/master'

    @priitkaard
    priitkaard committed Feb 10, 2022
    Configuration menu
    Copy the full SHA
    702e086 View commit details
    Browse the repository at this point in the history

  4. fix: tests

    @priitkaard
    priitkaard committed Feb 10, 2022
    Configuration menu
    Copy the full SHA
    f431356 View commit details
    Browse the repository at this point in the history

Commits on Feb 14, 2022

  1. revert: seek offsets data structure

    @priitkaard
    priitkaard committed Feb 14, 2022
    Configuration menu
    Copy the full SHA
    3580f9d View commit details
    Browse the repository at this point in the history

  2. fix: don't export error helper functions

    @priitkaard
    priitkaard committed Feb 14, 2022
    Configuration menu
    Copy the full SHA
    7342110 View commit details
    Browse the repository at this point in the history

  3. fix: tests and some types

    @priitkaard
    priitkaard committed Feb 14, 2022
    Configuration menu
    Copy the full SHA
    4a195d7 View commit details
    Browse the repository at this point in the history

  4. fix: use getConnection to get connection from pool

    @priitkaard
    priitkaard committed Feb 14, 2022
    Configuration menu
    Copy the full SHA
    49cc00c View commit details
    Browse the repository at this point in the history

  5. fix: update fetch event payloads

    @priitkaard
    priitkaard committed Feb 14, 2022
    Configuration menu
    Copy the full SHA
    d6f32b8 View commit details
    Browse the repository at this point in the history

  6. refactor: split runner and runnerPool tests

    @priitkaard
    priitkaard committed Feb 14, 2022
    Configuration menu
    Copy the full SHA
    0c7fa43 View commit details
    Browse the repository at this point in the history

Commits on Feb 15, 2022

  1. refactor: reduce coupling from runner to consumerGroup

    @priitkaard
    priitkaard committed Feb 15, 2022
    Configuration menu
    Copy the full SHA
    eef1431 View commit details
    Browse the repository at this point in the history

  2. refactor: assign fetch manager partitions on creating

    @priitkaard
    priitkaard committed Feb 15, 2022
    Configuration menu
    Copy the full SHA
    ec8e4d8 View commit details
    Browse the repository at this point in the history

  3. refactor: make consumerGroup.fetch private

    @priitkaard
    priitkaard committed Feb 15, 2022
    Configuration menu
    Copy the full SHA
    10c0e70 View commit details
    Browse the repository at this point in the history

  4. rename: message to fetchResult

    @priitkaard
    priitkaard committed Feb 15, 2022
    Configuration menu
    Copy the full SHA
    31176a5 View commit details
    Browse the repository at this point in the history

  5. feat: worker, workerQueue, fetcher, fetchManager

    @priitkaard
    priitkaard committed Feb 15, 2022
    Configuration menu
    Copy the full SHA
    02a8e51 View commit details
    Browse the repository at this point in the history

Commits on Feb 16, 2022

  1. * refactor: reduce coupling between consumerGroup and concurrent work… 

    …ers.

* revert: runner partially
* refactor: workerQueue, fetcher to use entities from parameters for better testability
* remove: old fetchManager and tests
    @priitkaard
    priitkaard committed Feb 16, 2022
    Configuration menu
    Copy the full SHA
    f328d45 View commit details
    Browse the repository at this point in the history

Commits on Feb 17, 2022

  1. feat: handle new/inactive nodes from refreshing cluster metadata

    @priitkaard
    priitkaard committed Feb 17, 2022
    Configuration menu
    Copy the full SHA
    b392db9 View commit details
    Browse the repository at this point in the history

  2. refactor: ConnectionPool 

    * getConnection will connect lazily
* move auth logic from broker to connection
    @priitkaard
    priitkaard committed Feb 17, 2022
    Configuration menu
    Copy the full SHA
    26410ed View commit details
    Browse the repository at this point in the history

  3. fix: flat() to flatten()

    @priitkaard
    priitkaard committed Feb 17, 2022
    Configuration menu
    Copy the full SHA
    e3af849 View commit details
    Browse the repository at this point in the history

Commits on Feb 18, 2022

  1. minor cleanup

    @priitkaard
    priitkaard committed Feb 18, 2022
    Configuration menu
    Copy the full SHA
    c13c147 View commit details
    Browse the repository at this point in the history

Commits on Feb 21, 2022

  1. fix: remove isEmptyDueToFiltering from public Batch interface

    @priitkaard
    priitkaard committed Feb 21, 2022
    Configuration menu
    Copy the full SHA
    a40e72b View commit details
    Browse the repository at this point in the history

  2. refactor: concurrency logic 

    * one fetcher per broker
* one shared workerQueue
* independent pool of workers
    @priitkaard
    priitkaard committed Feb 21, 2022
    Configuration menu
    Copy the full SHA
    af1cb7a View commit details
    Browse the repository at this point in the history

  3. fix: disconnecting status as connected

    @priitkaard
    priitkaard committed Feb 21, 2022
    Configuration menu
    Copy the full SHA
    1c6f37a View commit details
    Browse the repository at this point in the history

Commits on Feb 22, 2022

  1. chore: skip workerQueue.push if !isRunning

    @priitkaard
    priitkaard committed Feb 22, 2022
    Configuration menu
    Copy the full SHA
    c79858f View commit details
    Browse the repository at this point in the history

  2. fix: test

    @priitkaard
    priitkaard committed Feb 22, 2022
    Configuration menu
    Copy the full SHA
    2542663 View commit details
    Browse the repository at this point in the history

Commits on Feb 23, 2022

  1. feat: commit offsets before rejoining the consumer group on rebalance 

    fix: recover from rebalance outside concurrent workers
refactor: commitOffsets should not rejoin consumerGroup
    @priitkaard
    priitkaard committed Feb 23, 2022
    Configuration menu
    Copy the full SHA
    2b4e62b View commit details
    Browse the repository at this point in the history

Commits on Mar 1, 2022

  1. fix: handle unknown_member_id from offsetCommit

    @priitkaard
    priitkaard committed Mar 1, 2022
    Configuration menu
    Copy the full SHA
    e3c1c0c View commit details
    Browse the repository at this point in the history

Commits on Mar 7, 2022

  1. Improve performance of assignment diff

    @Nevon
    Nevon committed Mar 7, 2022
    Configuration menu
    Copy the full SHA
    65a82c8 View commit details
    Browse the repository at this point in the history

  2. Ensure only one worker can operate on a topic-partition at any given … 

    …time
    @Nevon
    Nevon committed Mar 7, 2022
    Configuration menu
    Copy the full SHA
    1c4b02e View commit details
    Browse the repository at this point in the history

Commits on Mar 8, 2022

  1. Apply suggestions from code review 

    Co-authored-by: Priit Käärd <priitkaard123@gmail.com>
    @Nevon @priitkaard
    Nevon and priitkaard committed Mar 8, 2022
    Configuration menu
    Copy the full SHA
    8a4716f View commit details
    Browse the repository at this point in the history

  2. Handle partition reassignment in fetcher 

    If a partition is reassigned to a different broker, it's
possible that a fetcher currently has a batch for that
topic-partition in the worker queue. This would cause double
processing. To avoid this, the fetch manager now keeps track
of which fetcher is currently handling each topic-partition,
and the fetchers will filter out batches for any topic-partition
that is currently being processed by a different fetcher.
    @Nevon
    Nevon committed Mar 8, 2022
    Configuration menu
    Copy the full SHA
    f64263f View commit details
    Browse the repository at this point in the history

  3. Revert 1c4b02e 

    Removes batch filtering from worker, which is now moved up
to the fetch manager.
    @Nevon
    Nevon committed Mar 8, 2022
    Configuration menu
    Copy the full SHA
    a10806f View commit details
    Browse the repository at this point in the history

  4. Merge branch 'handle-partition-reassignment' of github.com:tulios/kaf… 

    …kajs into handle-partition-reassignment
    @Nevon
    Nevon committed Mar 8, 2022
    Configuration menu
    Copy the full SHA
    d0e790d View commit details
    Browse the repository at this point in the history

  5. Merge pull request #2 from tulios/handle-partition-reassignment 

    Handle partition reassignment
    @priitkaard
    priitkaard committed Mar 8, 2022
    Configuration menu
    Copy the full SHA
    6ea295c View commit details
    Browse the repository at this point in the history

Commits on Mar 9, 2022

  1. Merge branch 'master' into master

    @Nevon
    Nevon committed Mar 9, 2022
    Configuration menu
    Copy the full SHA
    5c0191c View commit details
    Browse the repository at this point in the history

Commits on Mar 15, 2022

  1. Merge branch 'master' into master

    @Nevon
    Nevon committed Mar 15, 2022
    Configuration menu
    Copy the full SHA
    c359518 View commit details
    Browse the repository at this point in the history