backup: create external storage per-backup reqeust.

[YuJuncen]

fixed #11112

What problem does this PR solve?

Issue Number: fixed #11112

Problem Summary:

What is changed and how it works?

What's Changed:

Create external storage at handle_backup_task.

Check List

Tests

• Manual test (add detailed scripts or steps below)
  For a workload with 30K tables, backing up to S3 storage:

with this PR	master

Release note

None

[ti-chi-bot]

[REVIEW NOTIFICATION]

This pull request has been approved by:

• 3pointer
• kennytm

To complete the pull request process, please ask the reviewers in the list to review by filling /cc @reviewer in the comment.
After your PR has acquired the required number of LGTMs, you can assign this pull request to the committer in the list by filling /assign @committer in the comment to help you merge this pull request.

The full list of commands accepted by this bot can be found here.

Reviewer can indicate their review by submitting an approval review.
Reviewer can cancel approval by submitting a request changes review.

[YuJuncen]

/sig migrate
/component backup-restore

[YuJuncen]

/cc 3pointer kennytm

[YuJuncen]

/run-test

[3pointer]

/run-br-integration-tests

[3pointer]

/run-integration-br-test

[kennytm]

/test

[YuJuncen]

/hold

Let's check what happens to S3 storage. 🤔
cc @kennytm

            // FIXME: download() should be an async fn, to allow BR to cancel
            // a download task.
            // Unfortunately, this currently can't happen because the S3Storage
            // is not Send + Sync. See the documentation of S3Storage for reason.

[kennytm]

For reference, the comment was about #7236, which has been taken out in #9415.

Unfortunately, YuJuncen has tested that applying this PR brings #7236 back. According to hyperium/hyper#2649, this is likely due to the use of block_on_external_io, which spawns and kills a fresh tokio runtime in every write call.

We think that refactoring ExternalStorage::write to actually return a future (i.e. making it async fn) is the easiest approach.

[2021/10/21 21:48:40.623 +08:00] [FATAL] [lib.rs:465] ["dispatch dropped without returning error"] [backtrace="   0: tikv_util::set_panic_hook::{{closure}}
   1: std::panicking::rust_panic_with_hook
             at /rustc/2faabf579323f5252329264cc53ba9ff803429a3/library/std/src/panicking.rs:626:17
   2: std::panicking::begin_panic::{{closure}}
   3: std::sys_common::backtrace::__rust_end_short_backtrace
   4: std::panicking::begin_panic
   5: <futures_util::future::try_future::MapOk<Fut,F> as core::future::future::Future>::poll
   6: hyper::client::client::Client<C,B>::send_request::{{closure}}
   7: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
   8: rusoto_core::request::http_client_dispatch::{{closure}}
   9: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
  10: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
  11: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
  12: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
  13: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
  14: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
  15: <aws::s3::S3Storage as cloud::blob::BlobStorage>::put
  16: backup::writer::Writer::save_and_build_file
  17: backup::writer::BackupWriter::save
  18: backup::endpoint::BackupRange::backup
  19: core::ops::function::FnOnce::call_once{{vtable.shim}}
  20: <yatp::task::callback::Runner as yatp::pool::runner::Runner>::handle
  21: std::sys_common::backtrace::__rust_begin_short_backtrace
  22: core::ops::function::FnOnce::call_once{{vtable.shim}}
  23: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/2faabf579323f5252329264cc53ba9ff803429a3/library/alloc/src/boxed.rs:1572:9
      <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
             at /rustc/2faabf579323f5252329264cc53ba9ff803429a3/library/alloc/src/boxed.rs:1572:9
      std::sys::unix::thread::Thread::new::thread_start
             at /rustc/2faabf579323f5252329264cc53ba9ff803429a3/library/std/src/sys/unix/thread.rs:91:17
  24: start_thread
  25: __clone
"] [location=/data4/juncen/.cargo/registry/src/github.com-1ecc6299db9ec823/hyper-0.14.11/src/client/conn.rs:316] [thread_name=bkwkr-1]

/lgtm cancel

[YuJuncen]

/run-br-integration-tests

[YuJuncen]

/run-integration-br-tests

[YuJuncen]

/unhold the i/o part of external storage is asynchronous.

[YuJuncen]

/hold cancel

[YuJuncen]

/run-integration-br-tests

[kennytm]

/test

https://ci.pingcap.net/blue/organizations/jenkins/tikv_ghpr_test/detail/tikv_ghpr_test/11050/pipeline = #9309?

[YuJuncen]

test_resolve::test_joint_confchange 🤔

/test

[YuJuncen]

test_resolve::test_joint_confchange => #11325

/test

[kennytm]

/merge

[ti-chi-bot]

@kennytm: It seems you want to merge this PR, I will help you trigger all the tests:

/run-all-tests

You only need to trigger /merge once, and if the CI test fails, you just re-trigger the test that failed and the bot will merge the PR for you after the CI passes.

If you have any questions about the PR merge process, please refer to pr process.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the ti-community-infra/tichi repository.

[ti-chi-bot]

This pull request has been accepted and is ready to merge.

Commit hash: f648407

[kennytm]

/test test_node_multiple_rollback_merge

[kennytm]

/test test_server_transfer_leader_safe

[kennytm]

/test test_sending_fail_with_net_error

[kennytm]

/test test_unsafe_recover_update_region + test_sending_fail_with_net_error

[kennytm]

seriously the number of retries to get it work is really frustrating.