Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore: replace blake2b implementation by golang.org/x/crypto #157

Merged
merged 3 commits into from Jun 22, 2022

Conversation

Jorropo
Copy link
Contributor

@Jorropo Jorropo commented Jun 2, 2022

In 2016-12-13 d8e61c69ab46ca38328da2f4995abaf93b252290 golang.org/x/crypto gained a blake2b AVX implementation.

There is also an AVX2 one.

It is faster than what we are using right now.

This commit allows for faster code and remove a dependency.

For future improvements (like NEON) /x/crypto seems more maintained (it isn't an archived repo).

benchmark              old ns/op     new ns/op     delta
BenchmarkSum128-12     252           140           -44.35%
BenchmarkSum1K-12      1221          986           -19.24%

benchmark              old MB/s     new MB/s     speedup
BenchmarkSum128-12     507.37       911.98       1.80x
BenchmarkSum1K-12      838.70       1038.43      1.24x

Copy link
Contributor

@marten-seemann marten-seemann left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This looks good to me, but I'll leave the final approval to someone who has more context with these hash functions.

version.json Outdated Show resolved Hide resolved
register/blake2/multihash_blake2.go Outdated Show resolved Hide resolved
@Jorropo Jorropo force-pushed the x/crypto branch 2 times, most recently from 2d9b6cf to fa6025f Compare June 2, 2022 16:12
@aschmahmann
Copy link
Contributor

@Kubuxu any issues with merging?

@Jorropo not blocking here, but it'd be nice if there was some verification that the hashes from the different blake2 libraries are the same. AFAICT we don't have any here, either a Go test or adding to https://github.com/multiformats/multihash/ and bumping the dependency here would be great.

@marten-seemann
Copy link
Contributor

not blocking here, but it'd be nice if there was some verification that the hashes from the different blake2 libraries are the same

That would need to live in a separate Go module though, otherwise the dependency would stay in go.mod. Or maybe a one-time test would be enough to convince us that the libraries are doing the same?

@Jorropo
Copy link
Contributor Author

Jorropo commented Jun 2, 2022

I'm surprised there is no blake2 tests, I'll add some, likely blake2's official test set.

@aschmahmann
Copy link
Contributor

That would need to live in a separate Go module though, otherwise the dependency would stay in go.mod. Or maybe a one-time test would be enough to convince us that the libraries are doing the same?

I was thinking that if you did a PR with the tests separately from this one we could merge that, and then rebase this PR on top of master and make sure if passes. Since the tests would be hitting test vectors that are constant it'd be fine.

@Jorropo
Copy link
Contributor Author

Jorropo commented Jun 2, 2022

I dont think we actually need to check boths, blake2 is a spec if an implementation isnt following it its wrong and that likely a big security issue.

The tests can just compare to well known values.

Copy link
Member

@Kubuxu Kubuxu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No issues from my side, I trust golang/x/crypto to implement Blake2 correctly.

@Jorropo
Copy link
Contributor Author

Jorropo commented Jun 3, 2022

/cc @aschmahmann I have added blake2b tests, tested both on the old and new implementation they both work.

RFM on my side

@Jorropo Jorropo force-pushed the x/crypto branch 2 times, most recently from b22b476 to 5e5dc16 Compare June 3, 2022 19:26
@Jorropo
Copy link
Contributor Author

Jorropo commented Jun 3, 2022

I trust golang/x/crypto to implement Blake2 correctly.

Fair enough, I belive the goal was more to ensure the new registering is done correctly.
Maybe one used to take bits now it takes byte sizes, or something weird like that.

@Jorropo
Copy link
Contributor Author

Jorropo commented Jun 16, 2022

@Kubuxu I have fixed the lint rule, could I pls get this merged so I can open the bump PR in go-ipfs ?

@Kubuxu
Copy link
Member

Kubuxu commented Jun 22, 2022

Hey, tests seem to fail on the CI.
Sorry for the delay, I didn't know you were waiting on me.

In 2016-12-13 d8e61c69ab46ca38328da2f4995abaf93b252290 golang.org/x/crypto gained a blake2b AVX implementation.

There is also an AVX2 one.

It is faster than what we are using right now.

This commit allows for faster code and remove a dependency.

For future improvements (like NEON) /x/crypto seems more maintained (it isn't an archived repo).

benchmark              old ns/op     new ns/op     delta
BenchmarkSum128-12     252           140           -44.35%
BenchmarkSum1K-12      1221          986           -19.24%

benchmark              old MB/s     new MB/s     speedup
BenchmarkSum128-12     507.37       911.98       1.80x
BenchmarkSum1K-12      838.70       1038.43      1.24x
@Jorropo
Copy link
Contributor Author

Jorropo commented Jun 22, 2022

@Kubuxu
Sorry for the tests I had fixed it locally it but forgot to push it.

I don't know if it's waiting on you, you are just the only one with merge permissions in the conversation afait.

@marten-seemann marten-seemann merged commit 89bf86c into multiformats:master Jun 22, 2022
@Jorropo
Copy link
Contributor Author

Jorropo commented Jun 22, 2022

Thx, sorry for pinging you Kubuxu, weird that Marten only show up as Contributor.

Jorropo added a commit to Jorropo/go-ipfs that referenced this pull request Jun 22, 2022
This remove github.com/minio/blake2b-simd and replace it with golang.org/x/crypto/blake2b which slightly faster and reduce the code surface.

See multiformats/go-multihash#157 for more info.
aschmahmann pushed a commit to ipfs/kubo that referenced this pull request Jun 22, 2022
This remove github.com/minio/blake2b-simd and replace it with golang.org/x/crypto/blake2b which slightly faster and reduce the code surface.

See multiformats/go-multihash#157 for more info.
hannahhoward pushed a commit to filecoin-project/kubo-api-client that referenced this pull request Jun 19, 2023
This remove github.com/minio/blake2b-simd and replace it with golang.org/x/crypto/blake2b which slightly faster and reduce the code surface.

See multiformats/go-multihash#157 for more info.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants