Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Tokio based walredo #2875

Closed
wants to merge 12 commits into from
Closed

Tokio based walredo #2875

wants to merge 12 commits into from

Commits on Dec 7, 2022

  1. walredo: tests for existing implementation 

    adds pageserver/fixtures for the page image.
    @koivunej
    koivunej committed Dec 7, 2022
    Configuration menu
    Copy the full SHA
    6be11d5 View commit details
    Browse the repository at this point in the history

  2. bench: stop using launch_walredo

    @koivunej
    koivunej committed Dec 7, 2022
    Configuration menu
    Copy the full SHA
    4f575b8 View commit details
    Browse the repository at this point in the history

  3. bench: do single threaded bench in current thread 

    also reuse threads between runs for less noise in profiling.
    @koivunej
    koivunej committed Dec 7, 2022
    Configuration menu
    Copy the full SHA
    3cf6f6e View commit details
    Browse the repository at this point in the history

  4. bench: flat sampling 

    less warnings.
    @koivunej
    koivunej committed Dec 7, 2022
    Configuration menu
    Copy the full SHA
    54ba906 View commit details
    Browse the repository at this point in the history

  5. bench: stop repeating the request 

    it seems that we get much more stable values and less warnings out of
criterion this way.
    @koivunej
    koivunej committed Dec 7, 2022
    Configuration menu
    Copy the full SHA
    dab77cf View commit details
    Browse the repository at this point in the history

  6. walredo: asyncify wal redo process 

    reimplements the older poll and std::process::Command based
implementation to work on tokio primitives and latch on to task_mgr
lifecycle management.

there is a minor gain in performance with the pipelined requests, which
is partly negated by tokio just being slower than the blocking poll. the
implementation will get faster with future tokio enhancements, such as
vectored writes.
    @koivunej
    koivunej committed Dec 7, 2022
    Configuration menu
    Copy the full SHA
    09c5f34 View commit details
    Browse the repository at this point in the history

  7. walredo: adjust created spans 

    now it should be the hierarchy:

- walredo ctrl per tenant (tenant_id)
  - walredo per process (pid)
    - stdin_task
    - stdout_task
    - stderr_task
    @koivunej
    koivunej committed Dec 7, 2022
    Configuration menu
    Copy the full SHA
    950b907 View commit details
    Browse the repository at this point in the history

  8. walredo: scale to zero 

    first step in multi process controller. this implementation scales down
to zero in X times the outer queue read timeout in stdin_task,
it is probably too fast.
    @koivunej
    koivunej committed Dec 7, 2022
    Configuration menu
    Copy the full SHA
    943e896 View commit details
    Browse the repository at this point in the history

  9. walredo: simple multiprocess controller (scale up)

    @koivunej
    koivunej committed Dec 7, 2022
    Configuration menu
    Copy the full SHA
    9ca9c5e View commit details
    Browse the repository at this point in the history

  10. walredo: simple multiprocess controller (external) 

    integrate the process controller.
    @koivunej
    koivunej committed Dec 7, 2022
    Configuration menu
    Copy the full SHA
    e4e6174 View commit details
    Browse the repository at this point in the history

  11. WIP: Process received GetPage requests in parallel

    @hlinnaka @koivunej
    hlinnaka authored and koivunej committed Dec 7, 2022
    Configuration menu
    Copy the full SHA
    9eec603 View commit details
    Browse the repository at this point in the history

  12. test: remove any walredo dir in snapshotting

    @koivunej
    koivunej committed Dec 7, 2022
    Configuration menu
    Copy the full SHA
    56c7d15 View commit details
    Browse the repository at this point in the history