Make `htlc_maximum_msat` a required field. #1519

tnull · 2022-06-06T12:55:13Z

As of lightning/bolts#996, htlc_maximum_msat will soon be a required field.

This PR implements this change, i.e., it removes the special de-/serialisation logic needed before.
Not sure if we want to do this right away though, since it breaks serialisation backwards compat. 🤷‍♂️

TheBlueMatt · 2022-06-06T13:54:45Z

Given we can use this to clean up the scoring as well (we'd no longer need a "default" channel capacity), I think we should do this sooner rather than later. Will do a real review this week.

lightning/src/ln/msgs.rs

tnull · 2022-06-09T15:24:53Z

AFAICT, benchmark CI check is failing due to said compat breakage.

TheBlueMatt · 2022-06-10T17:29:32Z

Right, so I think compat is actually fine here, the issue is we have some channels in our kinda-checked-in gossip store that are missing the field. So we should update the NetworkGraph decoder to do something similar to what we do for P2P where we just silently ignore channels with missing htlc-max values and drop them from the graph. We'll re-add them if our peer has an updated copy with an htlc-max.

TheBlueMatt · 2022-06-10T17:33:32Z

I think, in this PR or a followup, we can drop EffectiveCapacity::Unknown which will be nice.

tnull · 2022-06-13T10:15:22Z

Right, so I think compat is actually fine here, the issue is we have some channels in our kinda-checked-in gossip store that are missing the field. So we should update the NetworkGraph decoder to do something similar to what we do for P2P where we just silently ignore channels with missing htlc-max values and drop them from the graph. We'll re-add them if our peer has an updated copy with an htlc-max.

Grr, after playing around with it a bit more, it seems just skipping on decoding errors isn't a very good option here, since this messes with the alignment in the reader. Also, I think that we would discard a lot of updates received via gossip and/or during deserialization when we just switch to the new layout.

Maybe the best way forward would be to introduce a LegacyUnsignedChannelUpdate (somewhat in analogy to #1529) that uses the old decoding and can be converted to UnsignedChannelUpdate by setting htlc_maximum_msat to the channel capacity when it was Absent. This way, in case of a failure, we could retry decoding+convert from the legacy format, instead of just skipping the update?

TheBlueMatt · 2022-06-13T11:09:19Z

Grr, after playing around with it a bit more, it seems just skipping on decoding errors isn't a very good option here, since this messes with the alignment in the reader.

Oh? I'm not sure why this is an issue? The last_update_message field we should be fine with, its already length-prefixed. The htlc_maximum_msat field directly in ChannelUpdateInfo implies we'll need to break out of the impl_writeable_tlv_based macro and write it by hand, making it MaybeReadable instead of Readable, but I don't see why that can't work.

tnull · 2022-06-13T12:24:52Z

Oh? I'm not sure why this is an issue? The last_update_message field we should be fine with, its already length-prefixed. The htlc_maximum_msat field directly in ChannelUpdateInfo implies we'll need to break out of the impl_writeable_tlv_based macro and write it by hand, making it MaybeReadable instead of Readable, but I don't see why that can't work.

Ah, I believe this could work in order to solve the alignment issue: we could ensure that we don't error-out mid decoding, i.e., the required number of bytes is always read, even if the ChannelUpdateInfo has no htlc_maximum_msat present and hence would result in a None.
However, I'm still not fully sure I can follow: wouldn't making ChannelUpdateInfo a MaybeReadable kind of chain up to ChannelInfo etc pp.? A slew of consequences seem to come with that, not sure we want those?

Also, are we positive we simply want to drop legacy channel updates when there is no htlc_maximum_msat present?

TheBlueMatt · 2022-06-13T15:34:45Z

Ah, I believe this could work in order to solve the alignment issue: we could ensure that we don't error-out mid decoding, i.e., the required number of bytes is always read, even if the ChannelUpdateInfo has no htlc_maximum_msat present and hence would result in a None.

Yep, exactly, read the whole thing either way, but set ChannelInfo::one_to_two/ChannelInfo::two_to_one to None instead of the inner field being None.

However, I'm still not fully sure I can follow: wouldn't making ChannelUpdateInfo a MaybeReadable kind of chain up to ChannelInfo etc pp.? A slew of consequences seem to come with that, not sure we want those?

Yes, but one_to_two and two_to_one are already Optional, so we can just bubble up the None to there and leave it.

Also, are we positive we simply want to drop legacy channel updates when there is no htlc_maximum_msat present?

I kinda assume so? Like, the theory of making it required is that its ~fully deployed everywhere and that we don't need to accept gossip where its not set. If that's true, we should be happy to not use gossip when its not set :)

tnull · 2022-06-15T16:33:31Z

Rebased and just pushed an intermediary step towards implementing MaybeReadable for ChannelUpdateInfo. Currently, a lot of tests fail.

I played around with utilizing pre-existing macros, but most of them are full of ?s, which would end decoding and therefore lead to the alignment issue. However, just writing everything from scratch seems like a lot of redundant code. I'm currently kind of stuck, have to think about this some more. Would also appreciate input in which direction to take this.

FWIW, I'm considering implementing _noerror variants of read_tlv_fields! and everything below to make this a bit easier. But this also may be a bit overblown..

TheBlueMatt

Yea, it does become a good bit more verbose...not a lot we can do about it, at least not without adding more support to the macros.

lightning/src/routing/gossip.rs

tnull · 2022-06-15T18:05:37Z

Yea, it does become a good bit more verbose...not a lot we can do about it, at least not without adding more support to the macros.

Alright, thanks for the feedback, will proceed.

Another conundrum to be solved is that even when we make sure we do not stop decoding, we will do so according to the new layout, i.e., will probably still hit an alignment issue. Will try to read and discard the appropriate number of bytes after we fail decoding the htlc_maximum_msat.

Btw. wouldn't such a change of the serialization format warrant to increase SERIALIZATION_VERSION? Or, if not, what would a be a case in which we want to do that?

TheBlueMatt · 2022-06-15T18:43:33Z

Another conundrum to be solved is that even when we make sure we do not stop decoding, we will do so according to the new layout, i.e., will probably still hit an alignment issue. Will try to read and discard the appropriate number of bytes after we fail decoding the htlc_maximum_msat.

It shouldn't be an issue as long as you call through the tlv stream read macro. It should read all available TLVs.

Btw. wouldn't such a change of the serialization format warrant to increase SERIALIZATION_VERSION? Or, if not, what would a be a case in which we want to do that?

No, I don't think so, I think that's basically just when we want to break TLV compat, but this should still be fully backwards and forwards compatible.

tnull · 2022-06-15T18:52:13Z

It shouldn't be an issue as long as you call through the tlv stream read macro. It should read all available TLVs.

Ah, right, one more reason why it could pay off to go the macro route. 👍

No, I don't think so, I think that's basically just when we want to break TLV compat, but this should still be fully
backwards and forwards compatible.

Thanks, makes sense.

TheBlueMatt · 2022-06-15T19:00:34Z

Ah, right, one more reason why it could pay off to go the macro route. 👍

I believe as written currently it works just fine and reads all available bytes.

TheBlueMatt · 2022-06-24T17:54:50Z

Looks like this needs rebase as well now. Feel free to squash fixups when you do so.

TheBlueMatt · 2022-07-05T15:35:24Z

As of #1553 we actually now must ensure we ignore announcements that fail to deserialize as we've "soft-forked" serialization and if a node has an existing, but invalid, hostname field we'll refuse to read our entire network graph. Thus, we'll want to do that here and will need to land this in the next release.

tnull · 2022-07-05T17:38:02Z

As of #1553 we actually now must ensure we ignore announcements that fail to deserialize as we've "soft-forked" serialization and if a node has an existing, but invalid, hostname field we'll refuse to read our entire network graph. Thus, we'll want to do that here and will need to land this in the next release.

Yes, sorry for the delay here. I made some progress and hope to push some updates by the end of this week, early next week latest.

codecov-commenter · 2022-07-12T13:01:14Z

Codecov Report

Merging #1519 (8b86ed7) into main (5023ff0) will increase coverage by 0.30%.
The diff coverage is 96.29%.

@@            Coverage Diff             @@
##             main    #1519      +/-   ##
==========================================
+ Coverage   90.82%   91.13%   +0.30%     
==========================================
  Files          80       80              
  Lines       44643    46470    +1827     
  Branches    44643    46470    +1827     
==========================================
+ Hits        40547    42350    +1803     
- Misses       4096     4120      +24

Impacted Files	Coverage Δ
lightning/src/ln/channel.rs	`88.78% <ø> (-0.01%)`	⬇️
lightning/src/ln/onion_route_tests.rs	`97.68% <ø> (-0.01%)`	⬇️
lightning/src/ln/priv_short_conf_tests.rs	`96.59% <ø> (-0.01%)`	⬇️
lightning/src/routing/router.rs	`92.06% <ø> (-0.39%)`	⬇️
lightning/src/routing/scoring.rs	`96.09% <ø> (-0.01%)`	⬇️
lightning/src/util/test_utils.rs	`78.11% <ø> (-0.05%)`	⬇️
lightning/src/routing/gossip.rs	`91.80% <95.83%> (+0.47%)`	⬆️
lightning-rapid-gossip-sync/src/lib.rs	`90.90% <100.00%> (ø)`
lightning-rapid-gossip-sync/src/processing.rs	`91.40% <100.00%> (+0.96%)`	⬆️
lightning/src/ln/channelmanager.rs	`85.13% <100.00%> (+0.02%)`	⬆️
... and 15 more

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 5023ff0...8b86ed7. Read the comment docs.

tnull · 2022-07-12T13:06:11Z

Rebased and just pushed some progress. ChannelInfo and ChannelUpdateInfo are decoded and encoded just fine independently, but the MaybeReadable Option is still not behaving as it should when the two are combined. Currently figuring out a workaround, feels like I'm getting close(r) though.

TheBlueMatt

Not your bug, but due to #1553, whatever we do for ChannelUpdate we have to also do for NodeAnnouncement.

lightning/src/ln/msgs.rs

lightning/src/routing/gossip.rs

TheBlueMatt · 2022-07-12T22:49:37Z

lightning/src/routing/gossip.rs

@@ -715,16 +797,55 @@ impl fmt::Display for ChannelInfo {
 	}
 }

-impl_writeable_tlv_based!(ChannelInfo, {


Can we not just make ignorable work for impl_writeable_tlv_based?

Hum, it may have been possible before, but I think now that we handle reading ChannelUpdateInfos via ChannelUpdateInfoDeserWrap, we're needing a custom implementation of Readable for ChannelInfo anyways?

lightning-rapid-gossip-sync/src/processing.rs

tnull · 2022-07-14T11:51:00Z

I think, in this PR or a followup, we can drop EffectiveCapacity::Unknown which will be nice.

@TheBlueMatt If you don't mind, I'll do this in a follow-up PR.

TheBlueMatt

This basically LGTM, can you clean up the git history and I'll give it a proper review?

lightning/src/routing/gossip.rs

dunxen

ACK 7b637a3

This LGTM. Only did a light review of gossip.rs though.

TheBlueMatt · 2022-07-19T20:31:47Z

Oops needs rebase now as well.

tnull · 2022-07-20T07:33:29Z

Oops needs rebase now as well.

Rebased on main.

tnull · 2022-07-20T11:25:57Z

Can you look into fixing the benchmark CI failure?

Yes, I have a fix ready that overrides with the default if htlc_maximum_msat == u64::max_value(). However, I took away from our discussion above that we do not want this and rather want to use a fresh snapshot where the few remaining updates not setting the field are already left out on generation? I'll check in again with @arik-so on this.

TheBlueMatt · 2022-07-20T19:46:19Z

Ah, right. Yea, let's just move forward with this and merge it with the benchmark CI failing, IMO. We can fix it pre-release (or not, even, really).

TheBlueMatt · 2022-07-20T19:46:42Z

In any case, assigning @arik-so as a "please generate a new snapshot"

arik-so · 2022-07-20T19:48:13Z

Did my comment regarding the discussion with Rusty not send? I'm not seeing it in this thread, when I could have sworn I sent it. Either way, I sent @tnull a new snapshot for testing this morning.

lightning/src/ln/msgs.rs

lightning/src/routing/gossip.rs

wpaulino · 2022-07-20T21:49:52Z

lightning/src/routing/gossip.rs

+		// Check we can decode legacy ChannelInfo, even if the `two_to_one`/`one_to_two` fields
+		// fail to decode.


Let's add a similar test for NodeInfo missing announcement_info?

Ah, yes, good catch, was coming back to that. I now added the test 04b677c, but it's indeed currently still failing, as not all bytes of the invalid NodeAnnouncementInfo are directly read when it fails, they are only cleaned up by s.eat_remaining(), which however still leads to the read_tlv_fields! macro returning an InvalidValue I can't catch (see here).
Currently not entirely sure how I'll about it. As it's in fact the NetAddress that is failing to decode, I probably could implement MaybeReadable for Vec<NetAddress> which only adds the successful decodes to the resulting vector.

I now worked around this by introducing another NetAddressVecDeserWrapper in 285cb27, but this is really not pretty. It might be easier to switch NetAddress directly to MaybeReadable, since it is the part that may fail to decode. However, I think this would mess with serialization in other parts of the code, e.g., the Init message...

After some offline discussion with @TheBlueMatt I now went to ignore all errors arising from trying to decode NodeAnnouncementInfo in c1bfa3f, which eliminates the need for an additional NetAddressVecDeserWrapper.

tnull · 2022-07-21T08:05:18Z

Did my comment regarding the discussion with Rusty not send? I'm not seeing it in this thread, when I could have sworn I sent it. Either way, I sent @tnull a new snapshot for testing this morning.

Can confirm that benchmark passes with the new snapshot at least locally. Can we upload it, I'd then update the URL in benchmark CI and here before we merge this?

TheBlueMatt · 2022-07-21T16:22:49Z

Oh, oops, right, can you also add a test (and fix, I think its broken) reading a NodeAnnouncementInfo that contains Some() NodeAnnouncement that itself contains a invalid Hostname NetAddress (ie too short encoding and also invalid string?)? We have to handle such graphs as they can be saved today even if we won't write them in 0.0.110.

tnull · 2022-07-22T08:09:08Z

Can confirm that benchmark passes with the new snapshot at least locally. Can we upload it, I'd then update the URL in benchmark CI and here before we merge this?

Updated snapshot URL with c6ab815, benchmark now passes.

tnull · 2022-07-22T08:24:18Z

Oh, oops, right, can you also add a test (and fix, I think its broken) reading a NodeAnnouncementInfo that contains Some() NodeAnnouncement that itself contains a invalid Hostname NetAddress (ie too short encoding and also invalid string?)? We have to handle such graphs as they can be saved today even if we won't write them in 0.0.110.

As discussed offline, we now eat all errors arising from trying to read NodeAnnouncementInfo with c1bfa3f.

TheBlueMatt · 2022-07-22T17:11:11Z

lightning/src/routing/gossip.rs

+
+impl MaybeReadable for NodeAnnouncementInfoDeserWrapper {
+	fn read<R: io::Read>(reader: &mut R) -> Result<Option<Self>, DecodeError> {
+		loop {


Oh lol, uhhh, hmm, I don't think this quite works, but could. Calling read on a "normal' object may try to read several bytes at a time, which technically the Read implementation may refuse to do if it doesn't have enough bytes. Instead, if we fail to read once we should fall back to just calling reader.read() on an, eg, 4KB buffer repeatedly until it returns zero.

Ah, I had thought the loop would always make at least a u8 worth of progress since it would try to read the tlv_len: BigSize fields on every iteration. That said, I now adopted the copy approach from eat_remaining, which seems like the cleaner way to do it anyways.

TheBlueMatt

Patch otherwise LGTM.

wpaulino

Looks good, feel free to squash.

Fixes a deserialization incompatibility introduced with lightningdevkit#1553.

tnull · 2022-07-25T18:40:13Z

Looks good, feel free to squash.

Squashed!

TheBlueMatt reviewed Jun 6, 2022

View reviewed changes

lightning/src/ln/msgs.rs Show resolved Hide resolved

tnull force-pushed the 2022-06-require-htlc-max branch from 3fd4fdc to b24f081 Compare June 9, 2022 14:53

tnull force-pushed the 2022-06-require-htlc-max branch from b24f081 to 3e85de4 Compare June 15, 2022 16:32

TheBlueMatt reviewed Jun 15, 2022

View reviewed changes

lightning/src/routing/gossip.rs Show resolved Hide resolved

lightning/src/routing/gossip.rs Show resolved Hide resolved

lightning/src/routing/gossip.rs Show resolved Hide resolved

lightning/src/routing/gossip.rs Outdated Show resolved Hide resolved

TheBlueMatt added this to the 0.0.110 milestone Jul 5, 2022

TheBlueMatt mentioned this pull request Jul 11, 2022

BOLT 7: add gossip address descriptor type DNS hostname lightning/bolts#911

Merged

tnull force-pushed the 2022-06-require-htlc-max branch from 3e85de4 to e3fe348 Compare July 12, 2022 13:01

TheBlueMatt reviewed Jul 12, 2022

View reviewed changes

tnull commented Jul 14, 2022

View reviewed changes

lightning-rapid-gossip-sync/src/processing.rs Show resolved Hide resolved

TheBlueMatt reviewed Jul 14, 2022

View reviewed changes

lightning/src/routing/gossip.rs Outdated Show resolved Hide resolved

tnull force-pushed the 2022-06-require-htlc-max branch from 6d6ffd4 to 3d37831 Compare July 15, 2022 10:54

dunxen previously approved these changes Jul 19, 2022

View reviewed changes

tnull dismissed dunxen’s stale review via 24315e4 July 20, 2022 07:33

tnull force-pushed the 2022-06-require-htlc-max branch from 7b637a3 to 24315e4 Compare July 20, 2022 07:33

tnull force-pushed the 2022-06-require-htlc-max branch from c83955c to a979694 Compare July 20, 2022 10:12

TheBlueMatt assigned arik-so Jul 20, 2022

wpaulino reviewed Jul 20, 2022

View reviewed changes

wpaulino removed the Seeking Code Review label Jul 20, 2022

TheBlueMatt unassigned arik-so Jul 20, 2022

TheBlueMatt reviewed Jul 22, 2022

View reviewed changes

tnull force-pushed the 2022-06-require-htlc-max branch from c58be01 to 896f47b Compare July 25, 2022 07:20

wpaulino reviewed Jul 25, 2022

View reviewed changes

tnull added 3 commits July 25, 2022 20:35

Make htlc_maximum_msat a required field.

b0e8b73

Don't fail read NodeInfo for inv. NetAddress

8f4c951

Fixes a deserialization incompatibility introduced with lightningdevkit#1553.

Test serialization of ChannelInfo and NodeInfo

8b86ed7

tnull force-pushed the 2022-06-require-htlc-max branch from 896f47b to 8b86ed7 Compare July 25, 2022 18:39

wpaulino approved these changes Jul 25, 2022

View reviewed changes

TheBlueMatt approved these changes Jul 25, 2022

View reviewed changes

TheBlueMatt merged commit 1988cb2 into lightningdevkit:main Jul 25, 2022

		// Check we can decode legacy ChannelInfo, even if the `two_to_one`/`one_to_two` fields
		// fail to decode.

Make htlc_maximum_msat a required field. #1519

Make htlc_maximum_msat a required field. #1519

Conversation

tnull commented Jun 6, 2022

TheBlueMatt commented Jun 6, 2022

tnull commented Jun 9, 2022

TheBlueMatt commented Jun 10, 2022

TheBlueMatt commented Jun 10, 2022

tnull commented Jun 13, 2022 • edited

TheBlueMatt commented Jun 13, 2022

tnull commented Jun 13, 2022 • edited

TheBlueMatt commented Jun 13, 2022

tnull commented Jun 15, 2022 • edited

TheBlueMatt left a comment

Choose a reason for hiding this comment

tnull commented Jun 15, 2022 • edited

TheBlueMatt commented Jun 15, 2022

tnull commented Jun 15, 2022

TheBlueMatt commented Jun 15, 2022

TheBlueMatt commented Jun 24, 2022

TheBlueMatt commented Jul 5, 2022

tnull commented Jul 5, 2022 • edited

codecov-commenter commented Jul 12, 2022 • edited

Codecov Report

tnull commented Jul 12, 2022 • edited

TheBlueMatt left a comment

Choose a reason for hiding this comment

TheBlueMatt Jul 12, 2022

Choose a reason for hiding this comment

tnull Jul 14, 2022

Choose a reason for hiding this comment

tnull commented Jul 14, 2022 • edited

TheBlueMatt left a comment

Choose a reason for hiding this comment

dunxen left a comment

Choose a reason for hiding this comment

TheBlueMatt commented Jul 19, 2022

tnull commented Jul 20, 2022

tnull commented Jul 20, 2022

TheBlueMatt commented Jul 20, 2022

TheBlueMatt commented Jul 20, 2022

arik-so commented Jul 20, 2022

wpaulino Jul 20, 2022

Choose a reason for hiding this comment

tnull Jul 21, 2022

Choose a reason for hiding this comment

tnull Jul 21, 2022

Choose a reason for hiding this comment

tnull Jul 22, 2022 • edited

Choose a reason for hiding this comment

tnull commented Jul 21, 2022

TheBlueMatt commented Jul 21, 2022

tnull commented Jul 22, 2022

tnull commented Jul 22, 2022 • edited

TheBlueMatt Jul 22, 2022

Choose a reason for hiding this comment

tnull Jul 25, 2022 • edited

Choose a reason for hiding this comment

TheBlueMatt left a comment

Choose a reason for hiding this comment

wpaulino left a comment

Choose a reason for hiding this comment

tnull commented Jul 25, 2022

Make `htlc_maximum_msat` a required field. #1519

Make `htlc_maximum_msat` a required field. #1519

tnull commented Jun 13, 2022 •

edited

tnull commented Jun 13, 2022 •

edited

tnull commented Jun 15, 2022 •

edited

tnull commented Jun 15, 2022 •

edited

tnull commented Jul 5, 2022 •

edited

codecov-commenter commented Jul 12, 2022 •

edited

tnull commented Jul 12, 2022 •

edited

tnull commented Jul 14, 2022 •

edited

tnull Jul 22, 2022 •

edited

tnull commented Jul 22, 2022 •

edited

tnull Jul 25, 2022 •

edited