Merge pull request #3055 from TheBlueMatt/2024-05-123-backports-2

Final backports for 0.0.123

CHANGELOG.md

@@ -1,3 +1,160 @@
+# 0.0.123 - May 08, 2024 - "BOLT12 Dust Sweeping"
+
+## API Updates
+
+ * To reduce risk of force-closures and improve HTLC reliability the default
+   dust exposure limit has been increased to
+   `MaxDustHTLCExposure::FeeRateMultiplier(10_000)`. Users with existing
+   channels might want to consider using
+   `ChannelManager::update_channel_config` to apply the new default (#3045).
+ * `ChainMonitor::archive_fully_resolved_channel_monitors` is now provided to
+   remove from memory `ChannelMonitor`s that have been fully resolved on-chain
+   and are now not needed. It uses the new `Persist::archive_persisted_channel`
+   to inform the storage layer that such a monitor should be archived (#2964).
+ * An `OutputSweeper` is now provided which will automatically sweep
+   `SpendableOutputDescriptor`s, retrying until the sweep confirms (#2825).
+ * After initiating an outbound channel, a peer disconnection no longer results
+   in immediate channel closure. Rather, if the peer is reconnected before the
+   channel times out LDK will automatically retry opening it (#2725).
+ * `PaymentPurpose` now has separate variants for BOLT12 payments, which
+   include fields from the `invoice_request` as well as the `OfferId` (#2970).
+ * `ChannelDetails` now includes a list of in-flight HTLCs (#2442).
+ * `Event::PaymentForwarded` now includes `skimmed_fee_msat` (#2858).
+ * The `hashbrown` dependency has been upgraded and the use of `ahash` as the
+   no-std hash table hash function has been removed. As a consequence, LDK's
+   `Hash{Map,Set}`s no longer feature several constructors when LDK is built
+   with no-std; see the `util::hash_tables` module instead. On platforms that
+   `getrandom` supports, setting the `possiblyrandom/getrandom` feature flag
+   will ensure hash tables are resistant to HashDoS attacks, though the
+   `possiblyrandom` crate should detect most common platforms (#2810, #2891).
+ * `ChannelMonitor`-originated requests to the `ChannelSigner` can now fail and
+   be retried using `ChannelMonitor::signer_unblocked` (#2816).
+ * `SpendableOutputDescriptor::to_psbt_input` now includes the `witness_script`
+   where available as well as new proprietary data which can be used to
+   re-derive some spending keys from the base key (#2761, #3004).
+ * `OutPoint::to_channel_id` has been removed in favor of
+   `ChannelId::v1_from_funding_outpoint` in preparation for v2 channels with a
+   different `ChannelId` derivation scheme (#2797).
+ * `PeerManager::get_peer_node_ids` has been replaced with `list_peers` and
+   `peer_by_node_id`, which provide more details (#2905).
+ * `Bolt11Invoice::get_payee_pub_key` is now provided (#2909).
+ * `Default[Message]Router` now take an `entropy_source` argument (#2847).
+ * `ClosureReason::HTLCsTimedOut` has been separated out from
+   `ClosureReason::HolderForceClosed` as it is the most common case (#2887).
+ * `ClosureReason::CooperativeClosure` is now split into
+   `{Counterparty,Locally}Initiated` variants (#2863).
+ * `Event::ChannelPending::channel_type` is now provided (#2872).
+ * `PaymentForwarded::{prev,next}_user_channel_id` are now provided (#2924).
+ * Channel init messages have been refactored towards V2 channels (#2871).
+ * `BumpTransactionEvent` now contains the channel and counterparty (#2873).
+ * `util::scid_utils` is now public, with some trivial utilities to examine
+   short channel ids (#2694).
+ * `DirectedChannelInfo::{source,target}` are now public (#2870).
+ * Bounds in `lightning-background-processor` were simplified by using
+   `AChannelManager` (#2963).
+ * The `Persist` impl for `KVStore` no longer requires `Sized`, allowing for
+   the use of `dyn KVStore` as `Persist` (#2883, #2976).
+ * `From<PaymentPreimage>` is now implemented for `PaymentHash` (#2918).
+ * `NodeId::from_slice` is now provided (#2942).
+ * `ChannelManager` deserialization may now fail with `DangerousValue` when
+    LDK's persistence API was violated (#2974).
+
+## Bug Fixes
+ * Excess fees on counterparty commitment transactions are now included in the
+   dust exposure calculation. This lines behavior up with some cases where
+   transaction fees can be burnt, making them effectively dust exposure (#3045).
+ * `Future`s used as an `std::...::Future` could grow in size unbounded if it
+   was never woken. For those not using async persistence and using the async
+   `lightning-background-processor`, this could cause a memory leak in the
+   `ChainMonitor` (#2894).
+ * Inbound channel requests that fail in
+   `ChannelManager::accept_inbound_channel` would previously have stalled from
+   the peer's perspective as no `error` message was sent (#2953).
+ * Blinded path construction has been tuned to select paths more likely to
+   succeed, improving BOLT12 payment reliability (#2911, #2912).
+ * After a reorg, `lightning-transaction-sync` could have failed to follow a
+   transaction that LDK needed information about (#2946).
+ * `RecipientOnionFields`' `custom_tlvs` are now propagated to recipients when
+   paying with blinded paths (#2975).
+ * `Event::ChannelClosed` is now properly generated and peers are properly
+   notified for all channels that as a part of a batch channel open fail to be
+   funded (#3029).
+ * In cases where user event processing is substantially delayed such that we
+   complete multiple round-trips with our peers before a `PaymentSent` event is
+   handled and then restart without persisting the `ChannelManager` after having
+   persisted a `ChannelMonitor[Update]`, on startup we may have `Err`d trying to
+   deserialize the `ChannelManager` (#3021).
+ * If a peer has relatively high latency, `PeerManager` may have failed to
+   establish a connection (#2993).
+ * `ChannelUpdate` messages broadcasted for our own channel closures are now
+   slightly more robust (#2731).
+ * Deserializing malformed BOLT11 invoices may have resulted in an integer
+   overflow panic in debug builds (#3032).
+ * In exceedingly rare cases (no cases of this are known), LDK may have created
+   an invalid serialization for a `ChannelManager` (#2998).
+ * Message processing latency handling BOLT12 payments has been reduced (#2881).
+ * Latency in processing `Event::SpendableOutputs` may be reduced (#3033).
+
+## Node Compatibility
+ * LDK's blinded paths were inconsistent with other implementations in several
+   ways, which have been addressed (#2856, #2936, #2945).
+ * LDK's messaging blinded paths now support the latest features which some
+   nodes may begin relying on soon (#2961).
+ * LDK's BOLT12 structs have been updated to support some last-minute changes to
+   the spec (#3017, #3018).
+ * CLN v24.02 requires the `gossip_queries` feature for all peers, however LDK
+   by default does not set it for those not using a `P2PGossipSync` (e.g. those
+   using RGS). This change was reverted in CLN v24.02.2 however for now LDK
+   always sets the `gossip_queries` feature. This change is expected to be
+   reverted in a future LDK release (#2959).
+
+## Security
+0.0.123 fixes a denial-of-service vulnerability which we believe to be reachable
+from untrusted input when parsing invalid BOLT11 invoices containing non-ASCII
+characters.
+ * BOLT11 invoices with non-ASCII characters in the human-readable-part may
+   cause an out-of-bounds read attempt leading to a panic (#3054). Note that all
+   BOLT11 invoices containing non-ASCII characters are invalid.
+
+In total, this release features 150 files changed, 19307 insertions, 6306
+deletions in 360 commits since 0.0.121 from 17 authors, in alphabetical order:
+
+ * Arik Sosman
+ * Duncan Dean
+ * Elias Rohrer
+ * Evan Feenstra
+ * Jeffrey Czyz
+ * Keyue Bao
+ * Matt Corallo
+ * Orbital
+ * Sergi Delgado Segura
+ * Valentine Wallace
+ * Willem Van Lint
+ * Wilmer Paulino
+ * benthecarman
+ * jbesraa
+ * olegkubrakov
+ * optout
+ * shaavan
+
+
+# 0.0.122 - Apr 09, 2024 - "That Which Is Untested Is Broken"
+
+## Bug Fixes
+ * `Route` objects did not successfully round-trip through de/serialization
+   since LDK 0.0.117, which has now been fixed (#2897).
+ * Correct deserialization of unknown future enum variants. This ensures
+   downgrades from future versions of LDK do not result in read failures or
+   corrupt reads in cases where enums are written (#2969).
+ * When hitting lnd bug 6039, our workaround previously resulted in
+   `ChannelManager` persistences on every round-trip with our peer. These
+   useless persistences are now skipped (#2937).
+
+In total, this release features 4 files changed, 99 insertions, 55
+deletions in 6 commits from 1 author, in alphabetical order:
+ * Matt Corallo
+
+
 # 0.0.121 - Jan 22, 2024 - "Unwraps are Bad"
 
 ## Bug Fixes
@@ -17,6 +174,7 @@ deletions in 4 commits from 2 authors, in alphabetical order:
  * Jeffrey Czyz
  * Matt Corallo
 
+
 # 0.0.120 - Jan 17, 2024 - "Unblinded Fuzzers"
 
 ## API Updates
@@ -65,6 +223,7 @@ deletions in 79 commits from 9 authors, in alphabetical order:
  * optout
  * shuoer86
 
+
 # 0.0.119 - Dec 15, 2023 - "Spring Cleaning for Christmas"
 
 ## API Updates

fuzz/src/full_stack.rs

@@ -971,6 +971,8 @@ mod tests {
 
 		// create the funding transaction (client should send funding_created now)
 		ext_from_hex("0a", &mut test);
+		// Two feerate requests to check the dust exposure on the initial commitment tx
+		ext_from_hex("00fd00fd", &mut test);
 
 		// inbound read from peer id 1 of len 18
 		ext_from_hex("030112", &mut test);
@@ -1019,6 +1021,9 @@ mod tests {
 		// end of update_add_htlc from 0 to 1 via client and mac
 		ext_from_hex("ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff ab00000000000000000000000000000000000000000000000000000000000000 03000000000000000000000000000000", &mut test);
 
+		// Two feerate requests to check dust exposure
+		ext_from_hex("00fd00fd", &mut test);
+
 		// inbound read from peer id 0 of len 18
 		ext_from_hex("030012", &mut test);
 		// message header indicating message length 100
@@ -1040,6 +1045,8 @@ mod tests {
 
 		// process the now-pending HTLC forward
 		ext_from_hex("07", &mut test);
+		// Two feerate requests to check dust exposure
+		ext_from_hex("00fd00fd", &mut test);
 		// client now sends id 1 update_add_htlc and commitment_signed (CHECK 7: UpdateHTLCs event for node 03020000 with 1 HTLCs for channel 3f000000)
 
 		// we respond with commitment_signed then revoke_and_ack (a weird, but valid, order)
@@ -1115,6 +1122,9 @@ mod tests {
 		// end of update_add_htlc from 0 to 1 via client and mac
 		ext_from_hex("ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff ab00000000000000000000000000000000000000000000000000000000000000 03000000000000000000000000000000", &mut test);
 
+		// Two feerate requests to check dust exposure
+		ext_from_hex("00fd00fd", &mut test);
+
 		// now respond to the update_fulfill_htlc+commitment_signed messages the client sent to peer 0
 		// inbound read from peer id 0 of len 18
 		ext_from_hex("030012", &mut test);
@@ -1146,6 +1156,10 @@ mod tests {
 
 		// process the now-pending HTLC forward
 		ext_from_hex("07", &mut test);
+
+		// Two feerate requests to check dust exposure
+		ext_from_hex("00fd00fd", &mut test);
+
 		// client now sends id 1 update_add_htlc and commitment_signed (CHECK 7 duplicate)
 		// we respond with revoke_and_ack, then commitment_signed, then update_fail_htlc
 
@@ -1243,6 +1257,9 @@ mod tests {
 		// end of update_add_htlc from 0 to 1 via client and mac
 		ext_from_hex("ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff 5300000000000000000000000000000000000000000000000000000000000000 03000000000000000000000000000000", &mut test);
 
+		// Two feerate requests to check dust exposure
+		ext_from_hex("00fd00fd", &mut test);
+
 		// inbound read from peer id 0 of len 18
 		ext_from_hex("030012", &mut test);
 		// message header indicating message length 164
@@ -1264,6 +1281,8 @@ mod tests {
 
 		// process the now-pending HTLC forward
 		ext_from_hex("07", &mut test);
+		// Two feerate requests to check dust exposure
+		ext_from_hex("00fd00fd", &mut test);
 		// client now sends id 1 update_add_htlc and commitment_signed (CHECK 7 duplicate)
 
 		// connect a block with one transaction of len 125

lightning-background-processor/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "lightning-background-processor"
-version = "0.0.123-rc1"
+version = "0.0.123"
 authors = ["Valentine Wallace <vwallace@protonmail.com>"]
 license = "MIT OR Apache-2.0"
 repository = "https://github.com/lightningdevkit/rust-lightning"
@@ -22,11 +22,11 @@ default = ["std"]
 
 [dependencies]
 bitcoin = { version = "0.30.2", default-features = false }
-lightning = { version = "0.0.123-rc1", path = "../lightning", default-features = false }
-lightning-rapid-gossip-sync = { version = "0.0.123-rc1", path = "../lightning-rapid-gossip-sync", default-features = false }
+lightning = { version = "0.0.123", path = "../lightning", default-features = false }
+lightning-rapid-gossip-sync = { version = "0.0.123", path = "../lightning-rapid-gossip-sync", default-features = false }
 
 [dev-dependencies]
 tokio = { version = "1.35", features = [ "macros", "rt", "rt-multi-thread", "sync", "time" ] }
-lightning = { version = "0.0.123-rc1", path = "../lightning", features = ["_test_utils"] }
-lightning-invoice = { version = "0.31.0-rc1", path = "../lightning-invoice" }
-lightning-persister = { version = "0.0.123-rc1", path = "../lightning-persister" }
+lightning = { version = "0.0.123", path = "../lightning", features = ["_test_utils"] }
+lightning-invoice = { version = "0.31.0", path = "../lightning-invoice" }
+lightning-persister = { version = "0.0.123", path = "../lightning-persister" }

lightning-block-sync/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "lightning-block-sync"
-version = "0.0.123-rc1"
+version = "0.0.123"
 authors = ["Jeffrey Czyz", "Matt Corallo"]
 license = "MIT OR Apache-2.0"
 repository = "https://github.com/lightningdevkit/rust-lightning"
@@ -20,11 +20,11 @@ rpc-client = [ "serde_json", "chunked_transfer" ]
 [dependencies]
 bitcoin = "0.30.2"
 hex = { package = "hex-conservative", version = "0.1.1", default-features = false }
-lightning = { version = "0.0.123-rc1", path = "../lightning" }
+lightning = { version = "0.0.123", path = "../lightning" }
 tokio = { version = "1.35", features = [ "io-util", "net", "time", "rt" ], optional = true }
 serde_json = { version = "1.0", optional = true }
 chunked_transfer = { version = "1.4", optional = true }
 
 [dev-dependencies]
-lightning = { version = "0.0.123-rc1", path = "../lightning", features = ["_test_utils"] }
+lightning = { version = "0.0.123", path = "../lightning", features = ["_test_utils"] }
 tokio = { version = "1.35", features = [ "macros", "rt" ] }

lightning-custom-message/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "lightning-custom-message"
-version = "0.0.123-rc1"
+version = "0.0.123"
 authors = ["Jeffrey Czyz"]
 license = "MIT OR Apache-2.0"
 repository = "https://github.com/lightningdevkit/rust-lightning"
@@ -15,4 +15,4 @@ rustdoc-args = ["--cfg", "docsrs"]
 
 [dependencies]
 bitcoin = "0.30.2"
-lightning = { version = "0.0.123-rc1", path = "../lightning" }
+lightning = { version = "0.0.123", path = "../lightning" }

lightning-invoice/Cargo.toml

@@ -1,7 +1,7 @@
 [package]
 name = "lightning-invoice"
 description = "Data structures to parse and serialize BOLT11 lightning invoices"
-version = "0.31.0-rc1"
+version = "0.31.0"
 authors = ["Sebastian Geisler <sgeisler@wh2.tu-dresden.de>"]
 documentation = "https://docs.rs/lightning-invoice/"
 license = "MIT OR Apache-2.0"
@@ -21,13 +21,13 @@ std = ["bitcoin/std", "lightning/std", "bech32/std"]
 
 [dependencies]
 bech32 = { version = "0.9.0", default-features = false }
-lightning = { version = "0.0.123-rc1", path = "../lightning", default-features = false }
+lightning = { version = "0.0.123", path = "../lightning", default-features = false }
 secp256k1 = { version = "0.27.0", default-features = false, features = ["recovery", "alloc"] }
 serde = { version = "1.0.118", optional = true }
 bitcoin = { version = "0.30.2", default-features = false }
 
 [dev-dependencies]
-lightning = { version = "0.0.123-rc1", path = "../lightning", default-features = false, features = ["_test_utils"] }
+lightning = { version = "0.0.123", path = "../lightning", default-features = false, features = ["_test_utils"] }
 hex = { package = "hex-conservative", version = "0.1.1", default-features = false }
 serde_json = { version = "1"}
 hashbrown = { version = "0.13", default-features = false }

lightning-invoice/src/de.rs

@@ -43,7 +43,11 @@ mod hrp_sm {
 	}
 
 	impl States {
-		fn next_state(&self, read_symbol: char) -> Result<States, super::Bolt11ParseError> {
+		fn next_state(&self, read_byte: u8) -> Result<States, super::Bolt11ParseError> {
+			let read_symbol = match char::from_u32(read_byte.into()) {
+				Some(symb) if symb.is_ascii() => symb,
+				_ => return Err(super::Bolt11ParseError::MalformedHRP),
+			};
 			match *self {
 				States::Start => {
 					if read_symbol == 'l' {
@@ -119,7 +123,7 @@ mod hrp_sm {
 			*range = Some(new_range);
 		}
 
-		fn step(&mut self, c: char) -> Result<(), super::Bolt11ParseError> {
+		fn step(&mut self, c: u8) -> Result<(), super::Bolt11ParseError> {
 			let next_state = self.state.next_state(c)?;
 			match next_state {
 				States::ParseCurrencyPrefix => {
@@ -158,7 +162,7 @@ mod hrp_sm {
 
 	pub fn parse_hrp(input: &str) -> Result<(&str, &str, &str), super::Bolt11ParseError> {
 		let mut sm = StateMachine::new();
-		for c in input.chars() {
+		for c in input.bytes() {
 			sm.step(c)?;
 		}
 

lightning-net-tokio/Cargo.toml

@@ -1,6 +1,6 @@
 [package]
 name = "lightning-net-tokio"
-version = "0.0.123-rc1"
+version = "0.0.123"
 authors = ["Matt Corallo"]
 license = "MIT OR Apache-2.0"
 repository = "https://github.com/lightningdevkit/rust-lightning/"
@@ -16,9 +16,9 @@ rustdoc-args = ["--cfg", "docsrs"]
 
 [dependencies]
 bitcoin = "0.30.2"
-lightning = { version = "0.0.123-rc1", path = "../lightning" }
+lightning = { version = "0.0.123", path = "../lightning" }
 tokio = { version = "1.35", features = [ "rt", "sync", "net", "time" ] }
 
 [dev-dependencies]
 tokio = { version = "1.35", features = [ "macros", "rt", "rt-multi-thread", "sync", "net", "time" ] }
-lightning = { version = "0.0.123-rc1", path = "../lightning", features = ["_test_utils"] }
+lightning = { version = "0.0.123", path = "../lightning", features = ["_test_utils"] }