Upstream and fix preflight probing

[tnull]

Fixes #2525.
Closes #2499.
Closes #2517.

Currently still WIP and hence in draft.

TODO:

• Probe only up to the last public hop if there's no other choices #2517
• Add units tests for probing in general and for the liquidity-stealing case in particular.

[tnull]

Rebased to resolve minor conflicts and undrafted.

[codecov-commenter]

Codecov Report

Patch coverage: 88.07% and project coverage change: +0.46% 🎉

Comparison is base (9f3e127) 90.60% compared to head (f75ac9a) 91.07%.
Report is 43 commits behind head on main.

❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #2534      +/-   ##
==========================================
+ Coverage   90.60%   91.07%   +0.46%     
==========================================
  Files         112      113       +1     
  Lines       58860    64894    +6034     
  Branches    58860    64894    +6034     
==========================================
+ Hits        53332    59100    +5768     
- Misses       5528     5794     +266     

Files Changed	Coverage Δ
lightning-background-processor/src/lib.rs	85.89% <ø> (+4.48%)	⬆️
lightning/src/ln/onion_utils.rs	93.80% <ø> (+2.02%)	⬆️
lightning/src/routing/scoring.rs	95.71% <0.00%> (+1.43%)	⬆️
lightning-invoice/src/payment.rs	85.90% <50.00%> (-2.37%)	⬇️
lightning/src/ln/outbound_payment.rs	92.40% <66.66%> (+1.70%)	⬆️
lightning/src/ln/channelmanager.rs	91.59% <84.74%> (+3.67%)	⬆️
lightning/src/ln/payment_tests.rs	97.48% <98.38%> (+0.02%)	⬆️
lightning/src/ln/functional_tests.rs	98.11% <100.00%> (ø)
lightning/src/routing/router.rs	94.50% <100.00%> (+0.01%)	⬆️

... and 15 files with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[TheBlueMatt]

Feel free to squash the fixup commits, I think this LGTM.

[jkczyz]

Mostly minor things. The commit message for ae8f536 could use details on the motivation.

[tnull]

Feel free to squash the fixup commits, I think this LGTM.

Squashed the fixups.

Mostly minor things. The commit message for ae8f536 could use details on the motivation.

Addressed by the following changes:

> git diff-tree -U2  6087affd 787fed49
diff --git a/lightning-invoice/src/payment.rs b/lightning-invoice/src/payment.rs
index 60d82b54..f5b20d87 100644
--- a/lightning-invoice/src/payment.rs
+++ b/lightning-invoice/src/payment.rs
@@ -34,6 +34,5 @@ use core::time::Duration;
 /// If you wish to use a different payment idempotency token, see [`pay_invoice_with_id`].
 pub fn pay_invoice<C: AChannelManager>(
-       invoice: &Bolt11Invoice, retry_strategy: Retry,
-       channelmanager: &C
+       invoice: &Bolt11Invoice, retry_strategy: Retry, channelmanager: &C
 ) -> Result<PaymentId, PaymentError>
 {
@@ -54,6 +53,5 @@ pub fn pay_invoice<C: AChannelManager>(
 /// See [`pay_invoice`] for a variant which uses the [`PaymentHash`] for the idempotency token.
 pub fn pay_invoice_with_id<C: AChannelManager>(
-       invoice: &Bolt11Invoice, payment_id: PaymentId, retry_strategy: Retry,
-       channelmanager: &C
+       invoice: &Bolt11Invoice, payment_id: PaymentId, retry_strategy: Retry, channelmanager: &C
 ) -> Result<(), PaymentError>
 {
@@ -72,6 +70,5 @@ pub fn pay_invoice_with_id<C: AChannelManager>(
 /// [`pay_zero_value_invoice_with_id`].
 pub fn pay_zero_value_invoice<C: AChannelManager>(
-       invoice: &Bolt11Invoice, amount_msats: u64, retry_strategy: Retry,
-       channelmanager: &C
+       invoice: &Bolt11Invoice, amount_msats: u64, retry_strategy: Retry, channelmanager: &C
 ) -> Result<PaymentId, PaymentError>
 {
diff --git a/lightning/src/ln/channelmanager.rs b/lightning/src/ln/channelmanager.rs
index f7950cb3..785595a8 100644
--- a/lightning/src/ln/channelmanager.rs
+++ b/lightning/src/ln/channelmanager.rs
@@ -3567,8 +3567,8 @@ where
                                                "Avoided sending payment probe all the way to last hop {} as it is likely unannounced.",
                                                last_path_hop.short_channel_id
-                                               );
+                                       );
                                        let final_value_msat = path.final_value_msat();
                                        path.hops.pop();
-                                       if let Some(new_last) = path.hops.iter_mut().last() {
+                                       if let Some(new_last) = path.hops.last_mut() {
                                                new_last.fee_msat += final_value_msat;
                                        }
diff --git a/lightning/src/routing/router.rs b/lightning/src/routing/router.rs
index 62d454cb..415e7b54 100644
--- a/lightning/src/routing/router.rs
+++ b/lightning/src/routing/router.rs
@@ -2500,5 +2500,5 @@ where L::Target: Logger {
                                network_graph.node(&hop.node_id).map_or(false, |hop_node|
                                        hop_node.channels.iter().any(|scid| network_graph.channel(*scid)
-                                                       .map_or(false, |c| c.as_directed_from(&prev_hop_node_id).is_some()))
+                                               .map_or(false, |c| c.as_directed_from(&prev_hop_node_id).is_some()))
                                        )
                        };

[TheBlueMatt]

Shouldn't this consider blinded paths somehow? Presumably the last hop towards a blinded path is always pub so we'll never hit this case anyway, but still.