Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[bug]: Fail to chain sync on testnet3 & mainnet errors relating to: script witness item is larger than the max allowed size #7002

Closed
rsafier opened this issue Oct 9, 2022 · 24 comments · Fixed by #7004
Labels
bug Unintended code behaviour P0 very high priority issue/PR, blocker on all others

Comments

@rsafier
Copy link

rsafier commented Oct 9, 2022

Background

Fail to chain sync on testnet3.
The node does show it is up to the tip of the chain , but remains unsynced due to errors

2022-10-09 18:42:39.886 [ERR] LNWL: Unable to process chain reorg: unable to get block 0000000000000032fcf519b61aad2b966348e3f2d27687b26277933cc9881965: readScript: script witness item is larger than the max allowed size [count 396669, max 11000]
....
2022-10-09 18:46:18.715 [ERR] LNWL: Unable to complete chain rescan: readScript: script witness item is larger than the max allowed size [count 396669, max 11000]
...

2022-10-09 21:36:23.272 [ERR] LNWL: Unable to process chain reorg: unable to get block 0000000000000014820a254fbcdccce582df2194014c1e3b3f5ecd9259fce663: readScript: script witness item is larger than the max allowed size [count 33970, max 11000]

Your environment

  • 0.15.0-beta commit=v0.15.0-beta
  • Linux ip-10-21-2-70 5.13.0-1031-aws #35~20.04.1-Ubuntu SMP Mon Jun 13 22:30:30 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
  • backend bitcoind v23.0.0

Steps to reproduce

Unknown

Expected behaviour

"synced_to_chain": true

Actual behaviour

"synced_to_chain": false with following errors in logs:

2022-10-09 18:42:39.886 [ERR] LNWL: Unable to process chain reorg: unable to get block 0000000000000032fcf519b61aad2b966348e3f2d27687b26277933cc9881965: readScript: script witness item is larger than the max allowed size [count 396669, max 11000]
....
2022-10-09 18:46:18.715 [ERR] LNWL: Unable to complete chain rescan: readScript: script witness item is larger than the max allowed size [count 396669, max 11000]
...

2022-10-09 21:36:23.272 [ERR] LNWL: Unable to process chain reorg: unable to get block 0000000000000014820a254fbcdccce582df2194014c1e3b3f5ecd9259fce663: readScript: script witness item is larger than the max allowed size [count 33970, max 11000]

@rsafier rsafier added bug Unintended code behaviour needs triage labels Oct 9, 2022
@RandyMcMillan
Copy link
Contributor

😬

@benthecarman
Copy link
Contributor

Looks like lnd/btcd has a bug in their taproot implementation. In BIP342:

Script size limit The maximum script size of 10000 bytes does not apply. Their size is only implicitly bounded by the block weight limit.[9]

I am surprised this wasn't caught by tests. Are they not using the static test vectors?

@HamishMacEwan
Copy link

Having this with LND Version v0.15.1 on mainnet:

2022-10-10 10:27:48.835 [INF] LTND: Waiting for chain backend to finish sync, start_height=757924
2022-10-10 10:27:49.535 [ERR] LNWL: Unable to complete chain rescan: readScript: script witness item is larger than the max allowed size [count 33970, max 11000]

@benthecarman
Copy link
Contributor

Got the same error on mainnet:

2022-10-09 16:56:50.707 [ERR] LNWL: Unable to process chain reorg: unable to get block 0000000000000000000400a35a007e223a7fb8a622dc7b5aa5eaace6824291fb: readScript: script witness item is larger than the max allowed size [count 33970, max 11000]

@rsafier
Copy link
Author

rsafier commented Oct 9, 2022

I also also seeing on mainnet v0.15.0:

2022-10-09 21:56:48.549 [ERR] LNWL: Unable to process chain reorg: unable to get block 0000000000000000000400a35a007e223a7fb8a622dc7b5aa5eaace6824291fb: readScript: script witness item is larger than the max allowed size [count 33970, max 11000]

@ExImpius
Copy link

ExImpius commented Oct 9, 2022

Same

@fotongit
Copy link

fotongit commented Oct 9, 2022

Same here. Why it suddenly happened to all of us?

Answer: a transaction came in the Blockchain that LND was not able to manage, thus all nodes went down at the same block

@hieblmi
Copy link
Collaborator

hieblmi commented Oct 9, 2022

See this on v0.15.1/mainnet


2022-10-09 21:56:55.488 [ERR] LNWL: Unable to process chain reorg: unable to get block 0000000000000000000400a35a007e223a7fb8a622dc7b5aa5eaace6824291fb: readScript: script witness item is larger than the max allowed size [count 33970, max 11000]

@benthecarman
Copy link
Contributor

This seems to be the tx that can't be parsed

https://mempool.space/tx/7393096d97bfee8660f4100ffd61874d62f9a65de9fb6acf740c4c386990ef73

@hieblmi
Copy link
Collaborator

hieblmi commented Oct 9, 2022

This seems to be the tx that can't be parsed

https://mempool.space/tx/7393096d97bfee8660f4100ffd61874d62f9a65de9fb6acf740c4c386990ef73

Posted about here: https://twitter.com/brqgoo/status/1579216353780957185

@bnonni
Copy link

bnonni commented Oct 9, 2022

Same here. Running on testnet. bitcoind v22.0, lnd v0.15.1.

2022-10-09 22:20:06.866 [INF] LTND: Waiting for chain backend to finish sync, start_height=2350195
2022-10-09 22:20:07.828 [INF] LNWL: Started rescan from block 0000000000000027e44f015d141df1aeff3794ac4b0b7303ca1039c6baa35a4c (height 2350075) for 0 addresses
2022-10-09 22:20:08.093 [ERR] LNWL: Unable to complete chain rescan: readScript: script witness item is larger than the max allowed size [count 396669, max 11000]

seeing this printed over and over in bitcoind debug.log

2022-10-09T22:30:27Z ThreadRPCServer method=getblockheader user=bitcoinrpc
2022-10-09T22:30:28Z ThreadRPCServer method=getblockchaininfo user=bitcoinrpc

@bnonni
Copy link

bnonni commented Oct 9, 2022

This seems to be the tx that can't be parsed
https://mempool.space/tx/7393096d97bfee8660f4100ffd61874d62f9a65de9fb6acf740c4c386990ef73

Posted about here: https://twitter.com/brqgoo/status/1579216353780957185

if this is a mainnet tx, why is it impacting nodes running on testnet?

@benthecarman
Copy link
Contributor

This seems to be the tx that can't be parsed
mempool.space/tx/7393096d97bfee8660f4100ffd61874d62f9a65de9fb6acf740c4c386990ef73

Posted about here: twitter.com/brqgoo/status/1579216353780957185

if this is a mainnet tx, why is it impacting nodes running on testnet?

A similar transcation happened on testnet

@rsafier rsafier changed the title [bug]: Fail to chain sync on testnet3 errors relating to: script witness item is larger than the max allowed size [bug]: Fail to chain sync on testnet3 & mainnet errors relating to: script witness item is larger than the max allowed size Oct 9, 2022
@bnonni
Copy link

bnonni commented Oct 9, 2022

This seems to be the tx that can't be parsed
mempool.space/tx/7393096d97bfee8660f4100ffd61874d62f9a65de9fb6acf740c4c386990ef73

Posted about here: twitter.com/brqgoo/status/1579216353780957185

if this is a mainnet tx, why is it impacting nodes running on testnet?

A similar transcation happened on testnet

makes sense. figured he must have tried it on testnet first. but the testnet block referenced doesn't appear to have a tx at the same scale as the one on mainnet
https://mempool.space/testnet/block/0000000000000027e44f015d141df1aeff3794ac4b0b7303ca1039c6baa35a4c

@spyhuntergenral
Copy link

ahh, few, thought i broke my node for a second .

@benthecarman
Copy link
Contributor

@bnonni
Copy link

bnonni commented Oct 9, 2022

Ahh, thanks. I see, its rescanning starting at block height 2350075 meaning the tx is in the next block. Got it. Well damn, I finally got my testnet node up-and-running and was about to make some taro assets and now this 😞

@jblachly
Copy link
Contributor

jblachly commented Oct 9, 2022

Bug is actually in btcd code:

https://github.com/btcsuite/btcd/blob/fc36cb25a4bdc6d989f9161552e7b0fe08b02939/wire/msgtx.go#L105-L109

@saubyk saubyk added the P0 very high priority issue/PR, blocker on all others label Oct 9, 2022
@brqgoo
Copy link

brqgoo commented Oct 9, 2022

With BIP-342, the maximum script size limit of 10000 bytes no longer applies. The witness script size is only implicitly bounded by the block weight limit.
https://github.com/bitcoin/bips/blob/master/bip-0342.mediawiki#cite_ref-9-0

@Roasbeef
Copy link
Member

Roasbeef commented Oct 9, 2022

Hey y'all, thanks for bringing this to our attention. I've identified the issue in the btcd wire parsing library, which led to this incident.

AFAICT, the consensus code wasn't the issue here, it was instead that the wire parsing library was erroneously still enforcing a prior check to limit witness sizing left over from segwit v0.

I am surprised this wasn't caught by tests. Are they not using the static test vectors?

btcd is/was using this test vectors. The issue here is that the code the parsed the witnesses for these test vectors isn't the same code that's used to read blocks off the wire. When a new block comes in, we fetch the raw block then attempt to decode it, which triggered this issue.

@indomitorum
Copy link

indomitorum commented Oct 9, 2022

Same problem. Have stopped the node now until further notice. 0.15.1

Logs when starting the node

Oct 10 00:34:36 indomitusbtc lnd[1922]: 2022-10-10 00:34:36.014 [INF] LTND: Waiting for chain backend to finish sync, start_height=757941

Oct 10 00:34:36 indomitusbtc lnd[1922]: 2022-10-10 00:34:36.399 [INF] LNWL: Started rescan from block 00000000000000000007477b90ff0ea53cdd1db88e03799af18ff58df0cebaa7 (height 757921) for 2056 addresses

Oct 10 00:34:36 indomitusbtc lnd[1922]: 2022-10-10 00:34:36.685 [ERR] LNWL: Unable to complete chain rescan: readScript: script witness item is larger than the max allowed size [count 33970, max 11000]

@Roasbeef
Copy link
Member

Roasbeef commented Oct 10, 2022

The fix to the btcd wire parsing logic can be found here: btcsuite/btcd#1896

This should be safe to apply to those running btcd nodes, which'll allow them to resume validating the main chain (the block was accepted as this wasn't a consensus issue persay).

Once this passes CI and a few more sniff checks, we'll issue a hotfix release for lnd: 0.15.2. This release will only contain the dependency update to the wire parsing library.

@chappjc
Copy link

chappjc commented Oct 10, 2022

Is a bump for neutrino needed too? I think a consumer can hoist the required version themselves, but they might have to require btc directly to do so.

@okeygo
Copy link

okeygo commented Oct 10, 2022

I confirm that my LND nodes are stuck at block 757921 despite I can send and receive Stas normally through established channels, I am proceeding to implement fix.

2022-10-10 08:19:21.786 [ERR] LNWL: Unable to process chain reorg: unable to get block 0000000000000000000400a35a007e223a7fb8a622dc7b5aa5eaace6824291fb: readScript: script witness item is larger than the max allowed size [count 33970, max 11000]
2022-10-10 08:19:22.258 [ERR] LNWL: Unable to process chain reorg: unable to get block 0000000000000000000400a35a007e223a7fb8a622dc7b5aa5eaace6824291fb: readScript: script witness item is larger than the max allowed size [count 33970, max 11000]
2022-10-10 08:20:02.063 [INF] CRTR: Processed channels=0 updates=198 nodes=11 in last 1m0.000466268s
2022-10-10 08:20:22.643 [INF] DISC: Broadcasting 321 new announcements in 18 sub batches
2022-10-10 08:21:02.063 [INF] CRTR: Processed channels=0 updates=234 nodes=8 in last 59.999372778s

nepet added a commit to nepet/peerswap that referenced this issue Oct 10, 2022
We need a fast update to the newest lnd version as there was a
security issue detected.
lightningnetwork/lnd#7002

As the bug was part of the btcd dependencies the fix applied in
btcsuite/btcd#1896 brings a long tail
of dependency updates.

This commit updates the dependencies and adapts for all changes
that are introduced by this new dependencies.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Unintended code behaviour P0 very high priority issue/PR, blocker on all others
Projects
None yet
Development

Successfully merging a pull request may close this issue.