Add BIP152 (Compact Blocks) structures #249

stevenroose · 2019-03-24T18:16:00Z

Adds the basic structures for BIP152 and a method to create a compact block from a full block.

Depends on rust-bitcoin/bitcoin_hashes#46.

Cargo.toml

src/util/bip152.rs

stevenroose · 2019-04-26T14:35:39Z

So I understand we don't want a siphasher dependency, but rather a bitcoin_hashes::siphash impl?

tamasblummer · 2019-04-26T14:41:07Z

yes, siphash is also used in BIP158. It would be better included into bitcoin_hashes

src/util/bip152.rs

stevenroose · 2019-05-14T21:26:59Z

yes, siphash is also used in BIP158. It would be better included into bitcoin_hashes

@tamasblummer
I added a siphash24 module in bitcoin_hashes: rust-bitcoin/bitcoin_hashes#46

Could you check if that suits your need for BIP 158 usage? I'm actually very interested in BIP-158 implementation in rust-bitcoin. Are you planning to do them?

tamasblummer · 2019-05-14T21:33:21Z

@stevenroose will review tomorrow. I already implemented and using BIP158 in murmel: https://github.com/rust-bitcoin/murmel/blob/master/src/blockfilter.rs

stevenroose · 2019-05-17T16:38:57Z

I already implemented and using BIP158 in murmel: https://github.com/rust-bitcoin/murmel/blob/master/src/blockfilter.rs

@tamasblummer would you mind upstreaming that to rust-bitcoin? Into src/util/bip185.rs f.e.?

stevenroose · 2019-05-30T11:02:21Z

@tamasblummer I made 2 changes:

HeaderAndShortIds::from_block now takes a version number instead of the use_wtxid bool. It returns an error if the version is not 1 or 2. This is probably better for backwards compatibility when we add support for other versions.
HeaderAndShortIds::from_block now takes an additional &[bool] argument to indicate which transactions to prefill. It must be of length one less than the nb of txs in the block because the coinbase is always prefilled. This is quite rudimentary support for arbitrary fillings. I thought about a closure argument Fn(&sha256d::Hash) -> bool. One could pass a HashSet::contains or HashMap::contains_key into that argument, f.e., as well as a custom method. Would that be a better solution? Thought? @apoelstra @dongcarl

stevenroose · 2019-07-26T13:37:34Z

This is rebased and ready to be reviewed. Uses the new bitcoin_hashes siphash impl.

src/network/message_compact_blocks.rs

stevenroose · 2019-10-10T14:51:28Z

rebased

stevenroose · 2021-11-15T01:04:40Z

@dr-orlovsky Addressed all your derives, had to derive PartialOrd, Ord, Hash on BlockHeader (which tbh I think makes more sense than on most of the compact block types), I did that in a separate commit.

vincenzopalazzo

ack 1d519df

dr-orlovsky

Concept ACK. I am not knowledgeable in compact filters to do a code vs standard review though :(

apoelstra · 2021-12-20T15:12:29Z

src/util/bip152.rs

+                            // strip witness for version 1
+                            let mut no_witness = tx.clone();
+                            no_witness.input.iter_mut().for_each(|i| i.witness.clear());
+                            no_witness


I know this PR is a million years old, but can you provide a citation/explanation for this witness-stripping behavior? I don't see it in Core or in BIP 152 or 158.

It's been ages indeed, but it says that txs inside v1 should be like responses to MSG_TX and v2 should be like responses to MSG_WITNESS_TX:
https://github.com/bitcoin/bips/blob/master/bip-0152.mediawiki#specification-for-version-2

apoelstra · 2021-12-20T15:16:01Z

Done review of 1d519df

ACK except for the thing about witness stripping.

I think this PR beats Carl's PSBT PR for "longest open PR" :)

stevenroose · 2022-03-10T17:43:36Z

Rebased once again! :)

tcharding · 2022-03-10T22:15:07Z

src/util/bip152.rs

+    }
+}
+
+impl hex::FromHex for ShortId {


Perhaps we should use impl_bytes_newtype to get this and the impls below?

tcharding · 2022-03-10T22:16:20Z

src/util/bip152.rs

+
+/// Short transaction IDs are used to represent a transaction without sending a full 256-bit hash.
+#[derive(PartialEq, Eq, Clone, Copy, Hash, Default, PartialOrd, Ord)]
+pub struct ShortId(pub [u8; 6]);


Should we have an AsRef implementation for ShortId as well?

0xB10C · 2022-04-04T08:12:01Z

Nice, was looking for the BIP152 network messages in rust-bitcoin and was about to implement them myself. Will test against Bitcoin Core's network messages and review!

0xB10C · 2022-04-05T08:42:14Z

src/util/bip152.rs

+    ///  The blockhash of the block which the transactions being requested are in.
+    pub block_hash: BlockHash,
+    ///  The indexes of the transactions being requested in the block.
+    pub indexes: Vec<u64>,


I guess using u64 is fine here though we'll probably not even going to get close to u32::MAX transactions in a block too.

tcharding · 2022-06-28T04:36:35Z

Hey mate, this PR is currently marked as part of the 0.29 milestone, would you like to get this functionality in before the 0.29 release?

0xB10C · 2022-06-29T22:28:08Z

I'm interested in seeing this in 0.29. I've rebased this branch on current master: https://github.com/0xB10C/rust-bitcoin/tree/compact-blocks - maybe this helps speed things up. I've been using 68760f5 to deserialize Bitcoin Core's compact block P2P messages for more than a month now without problems.

tcharding · 2022-06-29T22:36:34Z

Legend @0xB10C, thanks. Perhaps if @stevenroose does not pick this up in a week or so you could create a PR and just mention that its Steven's work and it supersedes this one [0]. I can do it if you either don't have time or don't feel comfortable doing so.

I'm totally speaking for @stevenroose here, Steven you can go right ahead and slap me if I'm out of place :)

[0]: When doing so I typically add a co-authored-by tag to any commit if I make any non-trivial changes.

0xB10C · 2022-07-06T15:37:44Z

src/util/bip152.rs

+                for _ in 0..nb_indexes {
+                    let differential: VarInt = Decodable::consensus_decode(&mut d)?;
+                    last_index  = match last_index.checked_add(differential.0 + 1) {
+                        Some(r) => r,
+                        None => return Err(encode::Error::ParseFailed("block index overflow")),
+                    };
+                    indexes.push(last_index);


for reference: While adding a deserialize(serialize(BlockTransactionsRequest{..})) test I noticed a subtle off-by-one error here. The increment in L321 (differential.0 + 1) is too early. This should only be incremented after we indexes.push(last_index);. Otherwise a differentially encoded [0, 1, 0] would become [1, 3, 4] when it should be [0, 2, 3].

Fixed this in #1088.

tcharding · 2022-07-18T04:30:46Z

Superseded by #1088

I've messaged @stevenroose via text also to make sure he is aware of this.

ee29910 BIP152: Test net msg ser/der and diff encoding (0xb10c) cd1aaaf BIP152: Add Compact Block unit test based on Elements (Steven Roose) d4d92a8 BIP152: Add Compact Blocks network messages (Steven Roose) f2fcdc8 BIP152: Add basic Compact Block structures (Steven Roose) a9a39c4 blockdata: Derive PartialOrd, Ord and Hash for BlockHeader (Steven Roose) Pull request description: > Adds the basic structures for BIP152 and a method to create a compact block from a full block. This is a rebase of #249 by stevenroose (see #249 (comment)) with a milestone for 29.0. I've added deserialization and serialization tests for the network messages and fixed an off-by-one bug in the deserialization of the differentially encoded varints in the `getblocktxn` message (see #249 (comment)). Closes #249. ACKs for top commit: tcharding: ACK ee29910 apoelstra: ACK ee29910 Tree-SHA512: 462a91576281f5a2ffdc2610769ea93970b60dac75a150c827966c48daec7cf93f526f9f202e7ba1dbb1410b49148579880260a3c3df298b98330c0d891a4cca

…uctures ee29910 BIP152: Test net msg ser/der and diff encoding (0xb10c) cd1aaaf BIP152: Add Compact Block unit test based on Elements (Steven Roose) d4d92a8 BIP152: Add Compact Blocks network messages (Steven Roose) f2fcdc8 BIP152: Add basic Compact Block structures (Steven Roose) a9a39c4 blockdata: Derive PartialOrd, Ord and Hash for BlockHeader (Steven Roose) Pull request description: > Adds the basic structures for BIP152 and a method to create a compact block from a full block. This is a rebase of #249 by stevenroose (see rust-bitcoin/rust-bitcoin#249 (comment)) with a milestone for 29.0. I've added deserialization and serialization tests for the network messages and fixed an off-by-one bug in the deserialization of the differentially encoded varints in the `getblocktxn` message (see rust-bitcoin/rust-bitcoin#249 (comment)). Closes #249. ACKs for top commit: tcharding: ACK ee29910 apoelstra: ACK ee29910 Tree-SHA512: 462a91576281f5a2ffdc2610769ea93970b60dac75a150c827966c48daec7cf93f526f9f202e7ba1dbb1410b49148579880260a3c3df298b98330c0d891a4cca

b3503ba Add example to SharedSecret (Tobin Harding) Pull request description: Currently the rustdoc on `SharedSecret` is wildly incorrect (possibly a cut'n'pasta error). Fix the rustdoc for `SharedSecret` and add an examples section to assist testing the public API. Fixes: rust-bitcoin#249 ACKs for top commit: apoelstra: ACK b3503ba Tree-SHA512: 650092388099bb415c11ea335ca6b64c90094f1a51ceecc403911316ee62da0279488af6fa66e00ee5269c129f06d4641085f8ab9be91c98d24a7a4449d235c2

Change freenode link to Libera

stevenroose changed the title ~~Implement BIP152 (Compact Blocks) structures~~ Add BIP152 (Compact Blocks) structures Mar 24, 2019

stevenroose force-pushed the compact-blocks branch from 990a31d to c8d0814 Compare April 1, 2019 15:50

stevenroose requested review from tamasblummer and dongcarl April 1, 2019 16:30

tamasblummer suggested changes Apr 1, 2019

View reviewed changes

Cargo.toml Outdated Show resolved Hide resolved

src/util/bip152.rs Outdated Show resolved Hide resolved

nothingmuch reviewed Apr 30, 2019

View reviewed changes

src/util/bip152.rs Outdated Show resolved Hide resolved

src/util/bip152.rs Outdated Show resolved Hide resolved

stevenroose force-pushed the compact-blocks branch from c8d0814 to 7a70807 Compare May 17, 2019 15:56

stevenroose changed the title ~~Add BIP152 (Compact Blocks) structures~~ WIP: Add BIP152 (Compact Blocks) structures May 17, 2019

stevenroose changed the title ~~WIP: Add BIP152 (Compact Blocks) structures~~ BLOCKED: Add BIP152 (Compact Blocks) structures May 17, 2019

stevenroose force-pushed the compact-blocks branch 2 times, most recently from 3a12d50 to 50717b1 Compare May 30, 2019 10:52

stevenroose force-pushed the compact-blocks branch from 50717b1 to a6beaed Compare May 30, 2019 11:05

stevenroose force-pushed the compact-blocks branch from a6beaed to 2ed6915 Compare July 6, 2019 22:46

stevenroose force-pushed the compact-blocks branch from 2ed6915 to 805aec6 Compare July 26, 2019 13:29

stevenroose changed the title ~~BLOCKED: Add BIP152 (Compact Blocks) structures~~ Add BIP152 (Compact Blocks) structures Jul 26, 2019

tamasblummer previously approved these changes Jul 26, 2019

View reviewed changes

TheBlueMatt reviewed Jul 26, 2019

View reviewed changes

src/network/message_compact_blocks.rs Show resolved Hide resolved

stevenroose dismissed tamasblummer’s stale review via f394fe0 July 26, 2019 16:11

stevenroose force-pushed the compact-blocks branch from 805aec6 to f394fe0 Compare July 26, 2019 16:11

tamasblummer previously approved these changes Aug 7, 2019

View reviewed changes

stevenroose dismissed tamasblummer’s stale review via b6ff8bb October 10, 2019 14:50

stevenroose force-pushed the compact-blocks branch from f394fe0 to b6ff8bb Compare October 10, 2019 14:50

stevenroose force-pushed the compact-blocks branch from ae56a06 to 1d519df Compare November 15, 2021 01:04

vincenzopalazzo previously approved these changes Nov 15, 2021

View reviewed changes

vincenzopalazzo mentioned this pull request Nov 15, 2021

Implementing BIP 152 cloudhead/nakamoto#70

Closed

dr-orlovsky reviewed Nov 15, 2021

View reviewed changes

apoelstra reviewed Dec 20, 2021

View reviewed changes

stevenroose added 4 commits March 10, 2022 17:30

blockdata: Derive PartialOrd, Ord and Hash for BlockHeader

ff14752

BIP152: Add basic Compact Block structures

1a06e74

BIP152: Add Compact Blocks network messages

2d5d6d1

BIP152: Add Compact Block unit test based on Elements

68760f5

stevenroose dismissed vincenzopalazzo’s stale review via 68760f5 March 10, 2022 17:43

stevenroose force-pushed the compact-blocks branch from 1d519df to 68760f5 Compare March 10, 2022 17:43

tcharding reviewed Mar 10, 2022

View reviewed changes

tcharding mentioned this pull request Mar 16, 2022

WIP: Add fuzz test for PrefilledTransaction #873

Closed

0xB10C reviewed Apr 5, 2022

View reviewed changes

0xB10C reviewed Jul 6, 2022

View reviewed changes

0xB10C mentioned this pull request Jul 6, 2022

Add BIP152 (Compact Blocks) structures #1088

Merged

tcharding closed this Jul 18, 2022

Kixunil removed this from the 0.29.0 milestone Aug 1, 2022

yancyribbens pushed a commit to yancyribbens/rust-bitcoin that referenced this pull request Mar 23, 2024

Merge pull request rust-bitcoin#249 from darosior/liberaforliberty

69d2d16

Change freenode link to Libera

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add BIP152 (Compact Blocks) structures #249

Add BIP152 (Compact Blocks) structures #249

stevenroose commented Mar 24, 2019 •

edited

stevenroose commented Apr 26, 2019

tamasblummer commented Apr 26, 2019

stevenroose commented May 14, 2019

tamasblummer commented May 14, 2019

stevenroose commented May 17, 2019

stevenroose commented May 30, 2019

stevenroose commented Jul 26, 2019

stevenroose commented Oct 10, 2019

stevenroose commented Nov 15, 2021

vincenzopalazzo left a comment

dr-orlovsky left a comment

apoelstra Dec 20, 2021

stevenroose Mar 10, 2022

apoelstra commented Dec 20, 2021

stevenroose commented Mar 10, 2022

tcharding Mar 10, 2022

tcharding Mar 10, 2022

0xB10C commented Apr 4, 2022

0xB10C Apr 5, 2022

tcharding commented Jun 28, 2022

0xB10C commented Jun 29, 2022

tcharding commented Jun 29, 2022

0xB10C Jul 6, 2022 •

edited

tcharding commented Jul 18, 2022

Add BIP152 (Compact Blocks) structures #249

Add BIP152 (Compact Blocks) structures #249

Conversation

stevenroose commented Mar 24, 2019 • edited

stevenroose commented Apr 26, 2019

tamasblummer commented Apr 26, 2019

stevenroose commented May 14, 2019

tamasblummer commented May 14, 2019

stevenroose commented May 17, 2019

stevenroose commented May 30, 2019

stevenroose commented Jul 26, 2019

stevenroose commented Oct 10, 2019

stevenroose commented Nov 15, 2021

vincenzopalazzo left a comment

Choose a reason for hiding this comment

dr-orlovsky left a comment

Choose a reason for hiding this comment

apoelstra Dec 20, 2021

Choose a reason for hiding this comment

stevenroose Mar 10, 2022

Choose a reason for hiding this comment

apoelstra commented Dec 20, 2021

stevenroose commented Mar 10, 2022

tcharding Mar 10, 2022

Choose a reason for hiding this comment

tcharding Mar 10, 2022

Choose a reason for hiding this comment

0xB10C commented Apr 4, 2022

0xB10C Apr 5, 2022

Choose a reason for hiding this comment

tcharding commented Jun 28, 2022

0xB10C commented Jun 29, 2022

tcharding commented Jun 29, 2022

0xB10C Jul 6, 2022 • edited

Choose a reason for hiding this comment

tcharding commented Jul 18, 2022

stevenroose commented Mar 24, 2019 •

edited

0xB10C Jul 6, 2022 •

edited