channel updates and node announcements not propagating

[JssDWt]

Issue and Steps to Reproduce

lightningd appears to fail to propagate its own channel updates through the network.
Yesterday, one of our nodes hadn't sent out a channel update or node announcement in over 22 days.
This corresponds with the latest restart of the node.
So it appears gossip is only exchanged when the node is restarted.
During this period there were several updates made to channel policies, but they also had not been propagated through the network.
Today the node was restarted and (some, but not all) gossip made it to other nodes.

I need some guidance how to troubleshoot this issue.

• Which logs should I look for?
• In which places in the codebase can I add logs to make it easier to troubleshoot?
• Are there settings we can use to increase gossip propagation?

getinfo output

{
   "id": "redacted",
   "alias": "redacted",
   "color": "redacted",
   "num_peers": 25,
   "num_pending_channels": 0,
   "num_active_channels": 31,
   "num_inactive_channels": 1,
   "address": [redacted],
   "binding": [redacted],
   "version": "v24.02.1",
   "blockheight": 840822,
   "network": "bitcoin",
   "fees_collected_msat": redacted,
   "lightning-dir": "redacted",
   "our_features": {
      "init": "08a0000a8a5961",
      "node": "88a0000a8a5961",
      "channel": "",
      "invoice": "02000002024100"
   }
}

[JssDWt]

I have gathered a view of other nodes on this node's channels, especially the last update. It appears 'some' gossip has been trickling in to other nodes, especially around 14-15 April, but very sparse. So it's not that all gossip has not propagated. Most of the gossip has not propagated.

Does cln periodically update the channel policy? Can you point me to the place in the code where that happens?

[urza]

I have also issue that my CLN channel updates are not propagating to the network.

In my case I traced it to issue with LND nodes in the network that mark some of my channels as "zombie" channels and refuse to propagate any gossip about them even if they are healthy channels.

It should hopefully be fixed by LND in 0.18 release.

lightningnetwork/lnd#8624

[JssDWt]

Querying the node announcement (with a little program written with LDK) from this node directly just now gives me a node announcement with the timestamp 1714019380, which is from April 24 (over a month ago). It appears the node announcement is not updated at all.

[vincenzopalazzo]

Mh! looking at this while thinking how to debug the following one lightningdevkit/rust-lightning#3075

I think @rustyrussell or @endothermicdev know the code base well, but if they do not have time I can start looking into it because at some point I should understand well gossipd anyway

[JssDWt]

Mh! looking at this while thinking how to debug the following one lightningdevkit/rust-lightning#3075

I customized the LDK program to just send a gossip query to the node and log the returned node announcement message of the node in question. It is really the only node announcement returned by the node and doesn't have to do with ordering of the messages.

[vincenzopalazzo]

Oh sorry I did not noted that you created a custom script to pool gossips :) I was talking about the general problem.

[JssDWt]

It does appear to be sending channel announcement first, then channel update, then node announcement though.

[JssDWt]

Looking at this a little bit more.
I have a node with 41 public channels.

If I query the gossip directly on the node itself, almost all channels have an update after 2024-05-28

$ lightning-cli listchannels source=02442d4249f9a93464aaf8cd8d522faa869356707b5f1537a8d6def2af50058c5b | jq '.channels[] | .last_update | strftime("%Y-%m-%dT%H:%M:%S %Z")'
"2024-05-28T18:19:06 UTC"
"2024-05-28T18:19:36 UTC"
"2024-05-30T10:09:58 UTC"
"2024-05-28T18:19:06 UTC"
"2024-05-28T18:19:06 UTC"
"2024-05-28T18:19:06 UTC"
"2024-05-28T18:19:06 UTC"
"2024-05-28T18:19:06 UTC"
"2024-05-28T18:19:06 UTC"
"2024-05-28T18:19:02 UTC"
"2024-05-28T18:19:02 UTC"
"2024-05-28T18:19:36 UTC"
"2024-05-28T18:19:03 UTC"
"2024-05-28T18:19:05 UTC"
"2024-05-28T18:19:06 UTC"
"2024-05-28T18:19:03 UTC"
"2024-05-28T18:19:06 UTC"
"2024-05-28T18:19:06 UTC"
"2024-05-28T18:19:06 UTC"
"2024-05-28T18:19:06 UTC"
"2024-05-28T18:19:06 UTC"
"2024-05-28T18:18:00 UTC"
"2024-05-28T18:18:00 UTC"
"2024-05-28T18:19:06 UTC"
"2024-05-28T18:19:06 UTC"
"2024-05-28T18:19:06 UTC"
"2024-05-28T18:19:06 UTC"
"2024-05-28T18:19:06 UTC"
"2024-05-29T18:57:07 UTC"
"2024-05-28T18:20:02 UTC"
"2024-05-28T18:19:06 UTC"
"2024-05-28T18:19:06 UTC"
"2024-05-31T21:49:11 UTC"
"2024-05-28T18:19:02 UTC"
"2024-05-28T18:19:03 UTC"
"2024-05-28T18:19:06 UTC"
"2024-05-28T18:19:06 UTC"
"2024-05-31T23:35:12 UTC"
"2024-05-28T18:19:03 UTC"
"2024-05-28T18:18:00 UTC"
"2024-05-18T17:58:17 UTC"
"2024-06-02T10:31:33 UTC"
"2024-06-02T10:31:33 UTC"

If I send a gossip timestamp filter to this node with first_timestamp == 0, I get all 41 public channels.
If, however, I send a gossip timestamp filter with first_timestamp == 7 days ago, which is before 2024-05-28, I would expect to get almost all the channels, but I get only 8 of its channels.
14 days I get 10 channels
28 days I get 17 channels
56 days I get 26 channels
112 days I get 35 channels
224 days I get all 41 channels

The oldest age of channels on the node is 127 days. So I think the opening timestamp is used to return channel updates, rather than the update date.