tikv: Fix the issue that KV requests might be processed slower because the connections to some KV servers are slow to establish.

[lysu]

What problem does this PR solve?

fix the potential problem that slow establishes the connection make other kv requests be slow

• 5 min dial timeout doesn't work in superbatch
• createConnArray will block wait conn establish on create tikvClient.BatchCommands
• long time hold lock in createConnArray will block getConnArray

What is changed and how it works?

• make establish grpc client in other goroutine which is blocking process
• check estDone before send request, send directly if done
• wait estReady channel if it's not done(only need select channel for first time)
• set right dialTimeout for batch client creation

more words:

• current impl is using connectivity state API, to wait non-blocking dial result with context timeout.
• new but not released grpc, we maybe can use client: add WithConnectParams to configure connection backoff and timeout grpc/grpc-go#2960 to simplify impl
• also take care Consider deprecating Dial and DialContext and replacing with a NewClient grpc/grpc-go#1786 if upgrade to new grpc in future

Check List

Tests

• Unit test
• Integration test

Code changes

• impl

Side effects

• n/a

Related changes

• Need to cherry-pick to the release branch

Release note

• Fix the issue that KV requests might be processed slower because the connections to some KV servers are slow to establish.

---

This change is 

[lysu]

/bench

[sre-bot]

@@                               Benchmark Diff                               @@
================================================================================
--- tidb: e80bab6d46869a77d56bdaef29d950bcbd4fdc40
+++ tidb: 7927cecc6c25a6843638795699c95d121c839165
tikv: 6b42cb67548e29d45e6f5e15da7ee9ba7099e993
pd: 381d36ba81d7616d8cd7e05c7672aa80a726f290
================================================================================
test-1: < oltp_point_select >
    * QPS : 139370.11 ± 0.7083% (std=626.58) delta: -0.19% (p=0.569)
    * AvgMs : 1.83 ± 0.8724% (std=0.01) delta: 0.49%
    * PercentileMs99 : 6.21 ± 0.0000% (std=0.00) delta: 0.00%
            
test-2: < oltp_read_write >
    * QPS : 41350.50 ± 1.7176% (std=415.34) delta: -0.58% (p=0.649)
    * AvgMs : 123.37 ± 0.1783% (std=0.14) delta: -0.16%
    * PercentileMs99 : 248.83 ± 0.0000% (std=0.00) delta: 0.00%
            
test-3: < oltp_insert >
    * QPS : 23847.62 ± 0.1959% (std=27.98) delta: -0.08% (p=0.782)
    * AvgMs : 10.73 ± 0.1864% (std=0.01) delta: 0.11%
    * PercentileMs99 : 22.05 ± 1.0614% (std=0.19) delta: 0.71%
            
test-4: < oltp_update_index >
    * QPS : 19240.50 ± 0.0552% (std=8.21) delta: 0.03% (p=0.551)
    * AvgMs : 13.26 ± 0.3771% (std=0.03) delta: -0.35%
    * PercentileMs99 : 30.81 ± 0.0000% (std=0.00) delta: 0.00%
            
test-5: < oltp_update_non_index >
    * QPS : 32242.45 ± 0.0556% (std=13.39) delta: -0.38% (p=0.091)
    * AvgMs : 7.94 ± 0.0840% (std=0.00) delta: 0.42%
    * PercentileMs99 : 20.74 ± 0.0000% (std=0.00) delta: 1.08%
            

https://perf.pingcap.com

[codecov]

Codecov Report

Merging #12733 into master will not change coverage.
The diff coverage is n/a.

@@             Coverage Diff             @@
##             master     #12733   +/-   ##
===========================================
  Coverage   80.0582%   80.0582%           
===========================================
  Files           465        465           
  Lines        106535     106535           
===========================================
  Hits          85290      85290           
  Misses        14897      14897           
  Partials       6348       6348

[lysu]

/run-all-tests

[lysu]

/bench

[sre-bot]

@@                               Benchmark Diff                               @@
================================================================================
--- tidb: 6b403e5767e117e599489070a495f7c3a7da787c
+++ tidb: b275614654d1a23e54ea9670f00957c5e9beede2
tikv: f7f9a5944feea92df87368874ca0575fb87cf9b1
pd: 543309d2613cbe3bc6592fa77e0c7e87692750c0
================================================================================
test-1: < oltp_point_select >
    * QPS : 138924.20 ± 0.1087% (std=96.15) delta: -1.06% (p=0.006)
    * AvgMs : 1.84 ± 0.0000% (std=0.00) delta: 0.99%
    * PercentileMs99 : 6.21 ± 0.0000% (std=0.00) delta: 1.97%
            
test-2: < oltp_read_write >
    * QPS : 41579.01 ± 0.2375% (std=61.29) delta: 0.18% (p=0.926)
    * AvgMs : 123.67 ± 0.2070% (std=0.18) delta: -0.01%
    * PercentileMs99 : 248.83 ± 0.0000% (std=0.00) delta: 1.08%
            
test-3: < oltp_insert >
    * QPS : 23718.83 ± 0.0252% (std=4.51) delta: -0.23% (p=0.377)
    * AvgMs : 10.79 ± 0.0618% (std=0.00) delta: 0.23%
    * PercentileMs99 : 22.28 ± 0.0000% (std=0.00) delta: 1.06%
            
test-4: < oltp_update_index >
    * QPS : 19155.67 ± 1.5763% (std=208.50) delta: -0.81% (p=0.482)
    * AvgMs : 13.24 ± 0.0755% (std=0.01) delta: -0.06%
    * PercentileMs99 : 30.81 ± 0.0000% (std=0.00) delta: 0.00%
            
test-5: < oltp_update_non_index >
    * QPS : 32284.11 ± 0.3777% (std=103.40) delta: 0.03% (p=0.865)
    * AvgMs : 7.92 ± 0.3281% (std=0.02) delta: -0.08%
    * PercentileMs99 : 20.52 ± 1.0820% (std=0.18) delta: 0.00%
            

https://perf.pingcap.com

[lysu]

PTAL @coocood @hicqu @tiancaiamao we need this in next release

[lysu]

/run-all-tests

[zyxbest]

/run-all-tests

[zyxbest]

/run-unit-tests

[coocood]

LGTM

[lysu]

/run-all-tests

[lysu]

/rebuild

[tiancaiamao]

LGTM

[lysu]

/run-all-tests

[lysu]

/rebuild

[zyxbest]

/run-all-tests

[lysu]

/bench

[sre-bot]

/run-all-tests

[sre-bot]

cherry pick to release-3.1 failed

[sre-bot]

cherry pick to release-3.0 failed