Better-behaved HBONE pooling (#931)

* Attempt at smarter HBONE pooling between ztunnels

Signed-off-by: Benjamin Leggett <benjamin.leggett@solo.io>

* Lints

Signed-off-by: Benjamin Leggett <benjamin.leggett@solo.io>

* lints 2

Signed-off-by: Benjamin Leggett <benjamin.leggett@solo.io>

* Hmm

Signed-off-by: Benjamin Leggett <benjamin.leggett@solo.io>

* Fixup

Signed-off-by: Benjamin Leggett <benjamin.leggett@solo.io>

* Fixup

Signed-off-by: Benjamin Leggett <benjamin.leggett@solo.io>

* More comments

Signed-off-by: Benjamin Leggett <benjamin.leggett@solo.io>

* cleanup

Signed-off-by: Benjamin Leggett <benjamin.leggett@solo.io>

* fixup

Signed-off-by: Benjamin Leggett <benjamin.leggett@solo.io>

* Clean

Signed-off-by: Benjamin Leggett <benjamin.leggett@solo.io>

* Fix jemalloc

Signed-off-by: Benjamin Leggett <benjamin.leggett@solo.io>

* WIP: move out of proxyinfo

Signed-off-by: Benjamin Leggett <benjamin.leggett@solo.io>

* Evict pooled conns after $INTERVAL

Signed-off-by: Benjamin Leggett <benjamin.leggett@solo.io>

* Update src/proxy/pool.rs

Co-authored-by: Ian Rudie <ilrudie@gmail.com>

* Evict pooled conns after $INTERVAL

Signed-off-by: Benjamin Leggett <benjamin.leggett@solo.io>

* For now, just do the foolproof collision check

Signed-off-by: Benjamin Leggett <benjamin.leggett@solo.io>

* Don't be silly

Signed-off-by: Benjamin Leggett <benjamin.leggett@solo.io>

* Naming, review comments

Signed-off-by: Benjamin Leggett <benjamin.leggett@solo.io>

* Tidy Arcs+drains

Signed-off-by: Benjamin Leggett <benjamin.leggett@solo.io>

* Cleanups

Signed-off-by: Benjamin Leggett <benjamin.leggett@solo.io>

* Format

Signed-off-by: Benjamin Leggett <benjamin.leggett@solo.io>

* Use the fancy lockless outer map, drop realm-io

Signed-off-by: Benjamin Leggett <benjamin.leggett@solo.io>

* Cleanup comments

Signed-off-by: Benjamin Leggett <benjamin.leggett@solo.io>

* Fix outdent (review comment)

Signed-off-by: Benjamin Leggett <benjamin.leggett@solo.io>

* Fixups/review comments

Signed-off-by: Benjamin Leggett <benjamin.leggett@solo.io>

* resync

Signed-off-by: Benjamin Leggett <benjamin.leggett@solo.io>

* Droptests

Signed-off-by: Benjamin Leggett <benjamin.leggett@solo.io>

* fix testhang

Signed-off-by: Benjamin Leggett <benjamin.leggett@solo.io>

* add smarter evict test

Signed-off-by: Benjamin Leggett <benjamin.leggett@solo.io>

* Interesting failure

Signed-off-by: Benjamin Leggett <benjamin.leggett@solo.io>

* No, it's not

Signed-off-by: Benjamin Leggett <benjamin.leggett@solo.io>

* Make this a bit simpler

Signed-off-by: Benjamin Leggett <benjamin.leggett@solo.io>

* Separate out the connspawner

Signed-off-by: Benjamin Leggett <benjamin.leggett@solo.io>

* Tidy logging a bit

Signed-off-by: Benjamin Leggett <benjamin.leggett@solo.io>

* Add serverside keepalive

Signed-off-by: Benjamin Leggett <benjamin.leggett@solo.io>

* fixup

Signed-off-by: Benjamin Leggett <benjamin.leggett@solo.io>

* Just for kicks

Signed-off-by: Benjamin Leggett <benjamin.leggett@solo.io>

* D'oh - use mthread runtime for tests

Signed-off-by: Benjamin Leggett <benjamin.leggett@solo.io>

* Fix none race

Signed-off-by: Benjamin Leggett <benjamin.leggett@solo.io>

* Propagate connection establish errors

Signed-off-by: Benjamin Leggett <benjamin.leggett@solo.io>

* Cleanup

Signed-off-by: Benjamin Leggett <benjamin.leggett@solo.io>

* Work around local test server getting overloaded

Signed-off-by: Benjamin Leggett <benjamin.leggett@solo.io>

* Move the rest to multi_thread, chill out on iterations, work around test
rig flakes

Signed-off-by: Benjamin Leggett <benjamin.leggett@solo.io>

* Tidy comments

Signed-off-by: Benjamin Leggett <benjamin.leggett@solo.io>

* lints

Signed-off-by: Benjamin Leggett <benjamin.leggett@solo.io>

* Clarify comment

Signed-off-by: Benjamin Leggett <benjamin.leggett@solo.io>

---------

Signed-off-by: Benjamin Leggett <benjamin.leggett@solo.io>
Co-authored-by: Ian Rudie <ilrudie@gmail.com>

Cargo.toml

@@ -97,6 +97,9 @@ url = "2.2"
 x509-parser = { version = "0.16", default-features = false }
 tracing-log = "0.2"
 backoff = "0.4.0"
+pin-project-lite = "0.2"
+pingora-pool = "0.1.0"
+flurry = "0.5.0"
 
 [target.'cfg(target_os = "linux")'.dependencies]
 netns-rs = "0.1"

src/admin.rs

@@ -408,7 +408,11 @@ fn change_log_level(reset: bool, level: &str) -> Response<Full<Bytes>> {
 async fn handle_jemalloc_pprof_heapgen(
     _req: Request<Incoming>,
 ) -> anyhow::Result<Response<Full<Bytes>>> {
-    let mut prof_ctl = jemalloc_pprof::PROF_CTL.as_ref()?.lock().await;
+    let mut prof_ctl = jemalloc_pprof::PROF_CTL
+        .as_ref()
+        .expect("should init")
+        .lock()
+        .await;
     if !prof_ctl.activated() {
         return Ok(Response::builder()
             .status(hyper::StatusCode::INTERNAL_SERVER_ERROR)
@@ -418,7 +422,7 @@ async fn handle_jemalloc_pprof_heapgen(
     let pprof = prof_ctl.dump_pprof()?;
     Ok(Response::builder()
         .status(hyper::StatusCode::OK)
-        .body(Bytes::from(pprof?).into())
+        .body(Bytes::from(pprof).into())
         .expect("builder with known status code should not fail"))
 }