btcec: create new btcec/v2 module that type aliases into the dcrec module #1773

Roasbeef · 2021-11-19T02:49:10Z

In this PR, we make the switch to remove most of the code from btcec in favor of creating a new module that uses type aliases to take advantage of all the constant timeness fixes and optimizations in the dcrec repo.

The changes were mostly mechanical, though there're a few things I need to fix like ensuring we still expose decoding signatures in the BER format.

Taking a look at the benchmarks, most operations other than normalization (which IIRC is a bit slower now due to constant time fixes) enjoy some nice speeds up:

benchcmp is deprecated in favor of benchstat: https://pkg.go.dev/golang.org/x/perf/cmd/benchstat
benchmark                            old ns/op     new ns/op     delta
BenchmarkAddJacobian-8               464           328           -29.20%
BenchmarkAddJacobianNotZOne-8        1138          372           -67.27%
BenchmarkScalarBaseMult-8            47336         31531         -33.39%
BenchmarkScalarBaseMultLarge-8       42465         32057         -24.51%
BenchmarkScalarMult-8                123355        117579        -4.68%
BenchmarkNAF-8                       582           168           -71.12%
BenchmarkSigVerify-8                 175414        120794        -31.14%
BenchmarkFieldNormalize-8            23.8          24.4          +2.39%
BenchmarkParseCompressedPubKey-8     24282         10907         -55.08%

Roasbeef · 2021-11-19T02:49:52Z

The PR doesn't build as is due to some Go modules oddities. I think we may actually need to make a new branch for the initial v2 switch over?

btcec/curve.go

davecgh · 2021-11-19T04:07:28Z

signature_test.go:345: trailing crap. signature failed when shouldn't malformed signature: bad length: 68 != 69

We don't allow this condition in Decred, as we enforce proper DER signature lengths. If memory serves, this must be allowed in BTC though and there are historical signatures on the chain that violate it.

I believe you'll have have to keep the btcec implementation slightly modified to accept the new types as a result.

Roasbeef · 2021-12-02T03:41:08Z

Pushed up a new version with things re-worked slightly:

We create a new v2 sub-folder where the new module lives. We need to do this due to the way btcutil is set up, since it uses btcec, and will not allow a new module creation otherwise.
Due to the above, we need to convert between types of the new and old module at times.

After going with this initial approach, thinking maybe things would be easier if we did it in the following order:

Add the new v2 sub folder
Update btcutil to use the new v2 module
Update btcd to use the new v2 module

Roasbeef · 2021-12-02T03:45:29Z

After the latest CI run, there's something off w/ sig parsing+verification still AFAICT. One peculiar thing is I had to remove the example (temporarily in an attempt to fix the module) as the sig validation was failing there.

As is, we use our existing signature parsing, but dcr's sig serialization, which afaict produces slightly stricter signatures.

coveralls · 2021-12-03T00:44:53Z

Pull Request Test Coverage Report for Build 1753757282

2 of 2 (100.0%) changed or added relevant lines in 1 file are covered.
No unchanged relevant lines lost coverage.
Overall coverage increased (+9.7%) to 89.423%

Totals
Change from base Build 1752974911:	9.7%
Covered Lines:	744
Relevant Lines:	832

💛 - Coveralls

Roasbeef · 2021-12-03T00:47:00Z

The build is now ✅!

Will proceed with the second leg now where I make a new PR to make a new ecdsa sub-folder, and add a BIP-340 implementation along side it.

I'm open to an alternative route vs the one laid out above, such as making this into a new v2 branch. Also important to note that the duplication is only temporary, as we can nuke the legacy version of btcec once a new btcec/v2 module is tagged here.

davecgh · 2021-12-03T00:49:54Z

Glad to hear it. Please retain the copyright headers from the files that were copied over from DCR.

Roasbeef · 2021-12-03T00:50:34Z

Also a note to reviewers: you'll want to check out the commits in order, so you can examine the diff due to the type alias port, and then the final API changes needed in the greater repo. In a few areas, I needed to convert types, since btcutil still uses the base v1 btcec package.

kcalvinalvin · 2021-12-03T07:03:59Z

After going with this initial approach, thinking maybe things would be easier if we did it in the following order:
1. Add the new v2 sub folder

2. Update btcutil to use the new v2 module

3. Update btcd to use the new v2 module

What about just moving btcutil into the btcd repo? There's really no reason for it to be a separate thing since it's so interconnected with btcd anyways. Looks like decred also did this as they have the dcrutil package inside the dcrd repo.

btcec/v2/README.md

integration/csv_fork_test.go

integration/rpcserver_test.go

integration/rpctest/rpc_harness_test.go

btcec/v2/bench_test.go

txscript/sign.go

kcalvinalvin · 2021-12-07T16:37:48Z

While reviewing, I saw that a lot of tests were removed as they were already being tested in dcrd anyways. This is an idea but if there isn't a strong objection against cgo, it'd be better to bind to Bitcoin core's secp256k1. Then there won't be any worries about bug incompatibility at the least.

davecgh · 2021-12-07T18:12:42Z

I'm no longer maintaining btcd, so it's not up to me, but using cgo would be an absolute 100% no from me if I were. If you care about maintaining quality software, binding crypto to cgo really is a bad idea in a project like this for several reasons. Among them:

it causes all kinds of cross compilation issues
it destroys the ability to create reproducible builds via the normal Go mechanisms
it is a constant source of bugs and suffers frequent run-time issues (look through the various Go patch releases)
it is difficult to review for correctness
calling into it has quite a lot of overhead and all of this code is used in super hot paths

All of that aside, I'd also note that the crypto itself is standardized, so the worry about compatibility is not in the core crypto itself, rather the potential compatibility issues arise in serialization and parsing due to Bitcoin's allowance of non-standard signature serializations. Pulling in an entire crypto library via cgo just to parse signatures seems like overkill, to say the least.

jcvernaleo · 2021-12-07T18:24:40Z

I agree with @davecgh there. I really think avoiding cgo is the way to go in nearly all cases.

guggero

Took me a moment to understand the logic behind the code move and the commit structure but I think it's quite nice to see the diff between the v1 and v2 code in the first commit.
Nice work preserving that!

LGTM assuming we fix the TODOs in the tests with the attached patch:
tests.patch.txt

I'm also very much against using cgo as that would break reproducible builds in lnd.

btcec/btcec_test.go

btcec/field_test.go

btcec/v2/README.md

integration/rpctest/memwallet.go

Roasbeef · 2022-01-11T22:27:25Z

Now that #1788 is merged, I'll pick this up again to get it properly building. We also should be able to remove one of the intermediate commits that just copies things over.

We'll also need to make sure we push out the proper git tags for the affected sub-modules afterwards.

Roasbeef · 2022-01-12T02:21:00Z

Just pushed up the new version! This is ready for another round of review now.

EDIT: looks like Actions is down? Got this on the first run: Error: Response status code does not indicate success: 503 (Service Unavailable).

Roasbeef · 2022-01-12T02:21:53Z

btcutil/address.go

@@ -469,9 +463,6 @@ func (a *AddressPubKey) serialize() []byte {

 	case PKFCompressed:
 		return a.pubKey.SerializeCompressed()
-
-	case PKFHybrid:
-		return a.pubKey.SerializeHybrid()


Ended up removing support for hybrid keys, as they don't exist in BIP 340 and have been non standard for some time elsewhere.

Roasbeef · 2022-01-12T02:22:13Z

btcutil/address_test.go

-			net: &chaincfg.MainNetParams,
-		},
-		{
-			name: "mainnet p2pk hybrid (0x07)",


Removed this since we drop support for serializing in a hybrid format.

Roasbeef · 2022-01-12T02:22:44Z

btcutil/hdkeychain/extendedkey.go

-	ilNum := new(big.Int).SetBytes(il)
-	if ilNum.Cmp(btcec.S256().N) >= 0 || ilNum.Sign() == 0 {
+	var ilNum btcec.ModNScalar
+	if overflow := ilNum.SetByteSlice(il); overflow {


Note that this is kinda optional, big.Ints work here, but wanted to make things more uniform so started to use the new ModNScalar.

Roasbeef · 2022-01-12T02:24:18Z

btcutil/hdkeychain/extendedkey.go

+		childKeyBytes := ilNum.Add(&keyNum).Bytes()
+		childKey = childKeyBytes[:]
+
+		// Strip leading zeroes from childKey, to match the expectation


The only odd thing I encountered during this translation, ran into something similar when getting the tests to pass for this original PR. As is, SetByteSlice assumes that all leading zeroes are stripped off, which is why it's added in the modified parseSig operation.

If this is removed, we fail TestLeadingZero.

btcutil/wif.go

integration/rpctest/memwallet.go

btcec/README.md

btcec/go.mod

btcec/ciphering.go

Roasbeef · 2022-01-24T03:49:26Z

Latest set of comments addressed. Should be g2g now, though I think the tool we use for coverages doesn't understand v2 modules?

/home/runner/work/_actions/shogo82148/actions-goveralls/v1/bin/goveralls_linux_amd64 -service=github -coverprofile=btcec/coverage.txt
Error: cannot read source of file "/home/runner/work/btcd/btcd/btcec/v2/btcec.go": open /home/runner/work/btcd/btcd/btcec/v2/btcec.go: no such file or directory
Error: Error: The process '/home/runner/work/_actions/shogo82148/actions-goveralls/v1/bin/goveralls_linux_amd64' failed with exit code 1

Roasbeef · 2022-01-26T21:53:52Z

Pushed new rebased version after the BIP 9 PR was merged. Once we get this in, I'll do a new tag for the v2 module, then rebase the schnorr PR as that'll be the final PR that needs to be merged for the main taproot PR to start building.

wpaulino

Should be g2g now, though I think the tool we use for coverages doesn't understand v2 modules?

It seems that the tool is using the go module path (btcec/v2/) instead of the directory path (btcec/). I wonder if moving all the contents of btcec/ into a subdir btcec/v2/ fixes it?

Makefile

Roasbeef · 2022-01-26T23:44:53Z

It seems that the tool is using the go module path (btcec/v2/) instead of the directory path (btcec/). I wonder if moving all the contents of btcec/ into a subdir btcec/v2/ fixes it?

If this is the case, then it's the tool that's messed up IMO. Just having to update the module means we don't need to actually move everything when/if we create a v3 version of the module.

I did a scan of the issue tracker for that repo and didn't see anything related to v2 modules....

Roasbeef · 2022-01-26T23:47:38Z

Perhaps there's some extra YAML field we need to be setting somewhere to tell it exactly where to look?

Roasbeef · 2022-01-26T23:56:47Z

Pushed a commit to try something out....

In this commit, we turn the package into a new Go module (version 2), and then port over the current set of types and functions to mainly alias to the more optimized and maintained dcrec variant. Taking a look at the benchmarks, most operations other than normalization (which IIRC is a bit slower now due to constant time fixes) enjoy some nice speeds up: ``` benchcmp is deprecated in favor of benchstat: https://pkg.go.dev/golang.org/x/perf/cmd/benchstat benchmark old ns/op new ns/op delta BenchmarkAddJacobian-8 464 328 -29.20% BenchmarkAddJacobianNotZOne-8 1138 372 -67.27% BenchmarkScalarBaseMult-8 47336 31531 -33.39% BenchmarkScalarBaseMultLarge-8 42465 32057 -24.51% BenchmarkScalarMult-8 123355 117579 -4.68% BenchmarkNAF-8 582 168 -71.12% BenchmarkSigVerify-8 175414 120794 -31.14% BenchmarkFieldNormalize-8 23.8 24.4 +2.39% BenchmarkParseCompressedPubKey-8 24282 10907 -55.08% ```

Roasbeef · 2022-01-27T00:12:30Z

Didn't work, but I think what's happening is that the coverage file itself has v2 in the path, so the tool attempts to grab the source to overlay/analyze the coverage (?) and that fails since there's no actual v2 path.

The `goveralls` tool we use to handle code coverage upload seems to not understand that a `v2 module can exist, without having a v2 file path on disk. We use a `sed` command to remove the `v2` module prefix so the tool can reach into the correct file to extract the source code.

Roasbeef · 2022-01-27T00:35:56Z

Great success: b3d263e

An alternative path would be updating the goveralls repo itself to understand the v2 module semantics, but this is quick and it does the job for now.

Crypt-iQ

Already wrote these, feel free to ignore since post-merge

Crypt-iQ · 2022-01-26T21:12:04Z

btcec/bench_test.go

+// This is NOT constant time.
+//
+// The field value is returned to support chaining.  This enables syntax like:
+// f := new(FieldVal).SetHex("0abc").Add(1) so that f = 0x0abc + 1


this comment seems slightly wrong since this function is setHex and not (* FieldVal).SetHex

Crypt-iQ · 2022-01-26T21:12:23Z

btcec/bench_test.go

+	if len(hexString)%2 != 0 {
+		hexString = "0" + hexString
+	}
+	bytes, _ := hex.DecodeString(hexString)


Is it assumed that only valid hex is passed in?

Crypt-iQ · 2022-01-26T21:14:22Z

btcec/bench_test.go

+}
+
+// fromHex converts the passed hex string into a big integer pointer and will
+// panic is there is an error.  This is only provided for the hard-coded


s/panic is/panic if
s/bet detected/be detected

Crypt-iQ · 2022-01-26T21:18:51Z

btcec/bench_test.go

+	return p
+}
+
+// BenchmarkAddNonConst benchmarks the secp256k1 curve AddNonConst function with


This seems to be the wrong godoc comment? If this is changed in a later commit I'll delete this

Crypt-iQ · 2022-01-26T21:47:11Z

btcec/btcec_test.go

+	tmp1.Mul(c1, endomorphismA1)
+	tmp2.Mul(c2, endomorphismA2)
+	k1.Sub(bigIntK, tmp1)
+	k1.Add(k1, tmp2)


it looks like this is doing:
k1 = k - c1 x a1 + c2 x a2
rather than what the comment says, am i missing something?

Crypt-iQ · 2022-01-26T21:49:29Z

btcec/btcec_test.go

+	// k2 = - c1 * b1 - c2 * b2 from step 5 (note c2's sign is reversed)
+	tmp1.Mul(c1, endomorphismB1)
+	tmp2.Mul(c2, endomorphismB2)
+	k2.Sub(tmp2, tmp1)


This also doesnt seem to match the comment which makes me think im missing context

Crypt-iQ · 2022-01-26T22:19:05Z

btcec/error.go

+)
+
+// Error identifies an error related to public key cryptography using a
+// sec256k1 curve. It has full support for errors.Is and errors.As, so the


s/sec256k1/secp256k1

Crypt-iQ · 2022-01-26T22:35:19Z

btcec/field_test.go

 			continue
 		}
 	}
 }

-// TestSquare ensures that squaring field values via Square works as expected.
-func TestSquare(t *testing.T) {
+// TestFieldSquare ensures that squaring field values via Square and SqualVal


s/SqualVal/SquareVal

Roasbeef force-pushed the dcr-ec branch from 6a301b4 to f2359d0 Compare November 19, 2021 02:55

davecgh reviewed Nov 19, 2021

View reviewed changes

btcec/curve.go Outdated Show resolved Hide resolved

Roasbeef force-pushed the dcr-ec branch from f2359d0 to 9db9e0d Compare November 19, 2021 03:52

Roasbeef force-pushed the dcr-ec branch from 9db9e0d to d604973 Compare December 2, 2021 03:34

Roasbeef force-pushed the dcr-ec branch 4 times, most recently from 2edaf4b to bdcd742 Compare December 3, 2021 00:41

Roasbeef mentioned this pull request Dec 5, 2021

btcec/v2: create new schnorr package for BIP-340, move existing ecdsa implementation into new ecdsa package #1777

Merged

2 tasks

kcalvinalvin reviewed Dec 7, 2021

View reviewed changes

guggero approved these changes Dec 9, 2021

View reviewed changes

btcec/btcec_test.go Show resolved Hide resolved

btcec/field_test.go Show resolved Hide resolved

btcec/field_test.go Outdated Show resolved Hide resolved

btcec/v2/README.md Outdated Show resolved Hide resolved

integration/rpctest/memwallet.go Outdated Show resolved Hide resolved

Roasbeef mentioned this pull request Dec 25, 2021

txscript: add taproot script type #1768

Closed

Roasbeef mentioned this pull request Jan 7, 2022

multi: implement BIP 341 and 342 a.k.a complete taproot and tapscript consensus verification logic #1787

Merged

5 tasks

Roasbeef force-pushed the dcr-ec branch from a3ff9aa to da2fc02 Compare January 12, 2022 02:20

Roasbeef commented Jan 12, 2022

View reviewed changes

Roasbeef force-pushed the dcr-ec branch 3 times, most recently from f54d599 to 878e182 Compare January 12, 2022 06:00

wpaulino reviewed Jan 12, 2022

View reviewed changes

Roasbeef force-pushed the dcr-ec branch from 878e182 to 8341f11 Compare January 24, 2022 03:43

Roasbeef requested a review from wpaulino January 24, 2022 03:43

Roasbeef force-pushed the dcr-ec branch from 8341f11 to 17d565d Compare January 26, 2022 21:54

wpaulino approved these changes Jan 26, 2022

View reviewed changes

Makefile Show resolved Hide resolved

Roasbeef added 4 commits January 26, 2022 16:10

multi: switch project over to using btcec/v2

eee3c3b

build: don't run the integration tests w/ -race

d2960c8

btcec/v2: retain copyright notices from decred developers for all files

3ee1211

Roasbeef force-pushed the dcr-ec branch from bc52d84 to 3ee1211 Compare January 27, 2022 00:10

Roasbeef merged commit a277387 into btcsuite:master Jan 27, 2022

Crypt-iQ reviewed Jan 27, 2022

View reviewed changes

btcec: create new btcec/v2 module that type aliases into the dcrec module #1773

btcec: create new btcec/v2 module that type aliases into the dcrec module #1773

Conversation

Roasbeef commented Nov 19, 2021 • edited

Roasbeef commented Nov 19, 2021

davecgh commented Nov 19, 2021

Roasbeef commented Dec 2, 2021

Roasbeef commented Dec 2, 2021

coveralls commented Dec 3, 2021 • edited

Pull Request Test Coverage Report for Build 1753757282

💛 - Coveralls

Roasbeef commented Dec 3, 2021

davecgh commented Dec 3, 2021

Roasbeef commented Dec 3, 2021

kcalvinalvin commented Dec 3, 2021

kcalvinalvin commented Dec 7, 2021

davecgh commented Dec 7, 2021 • edited

jcvernaleo commented Dec 7, 2021

guggero left a comment

Choose a reason for hiding this comment

Roasbeef commented Jan 11, 2022

Roasbeef commented Jan 12, 2022 • edited

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Roasbeef commented Jan 24, 2022

Roasbeef commented Jan 26, 2022

wpaulino left a comment

Choose a reason for hiding this comment

Roasbeef commented Jan 26, 2022

Roasbeef commented Jan 26, 2022

Roasbeef commented Jan 26, 2022

Roasbeef commented Jan 27, 2022 • edited

Roasbeef commented Jan 27, 2022

Crypt-iQ left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Roasbeef commented Nov 19, 2021 •

edited

coveralls commented Dec 3, 2021 •

edited

davecgh commented Dec 7, 2021 •

edited

Roasbeef commented Jan 12, 2022 •

edited

Roasbeef commented Jan 27, 2022 •

edited