Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add checksum validation on artifact upload #1063

Merged
merged 12 commits into from May 19, 2022
Merged

Conversation

robherley
Copy link
Member

@robherley robherley commented Apr 26, 2022

Relevant Issues:

tl;dr adds base64 encodings of the CRC64 and MD5 to the headers of a chunk upload:

x-actions-results-crc64
x-actions-results-md5

Recently we've been running into issues when artifacts are getting corrupted during the upload process. This is extremely rare (seen < 1% during my tests) but happens nonetheless. There still needs to be more investigation as to why these are being corrupted, but we've narrowed the problem areas to be between the upload and file container service handler after isolated testing with md5 checksum headers.

This PR adds a CRC64 and MD5 checksum to the header. Since NodeJS's crypto library (openssl bindings) doesn't have CRC64, I added a simple implementation based on Go's hash/crc64 pkg. Also this is tailored to the CRC polynomial used by azure storage. We're already using this same polynomial places deeper in the stack.

I ran some benchmarks and this CRC64 implementation is a bit faster than sha256/md5:

crc64 x 145,627 ops/sec ±8.33% (79 runs sampled)
md5 x 76,392 ops/sec ±12.44% (69 runs sampled)
sha256 x 65,982 ops/sec ±14.55% (62 runs sampled)
Fastest is crc64

@robherley robherley marked this pull request as ready for review May 19, 2022 16:06
@robherley robherley requested a review from a team as a code owner May 19, 2022 16:06
Copy link
Contributor

@yacaovsnc yacaovsnc left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

:shipit:

@robherley robherley merged commit a708045 into main May 19, 2022
bors bot added a commit to OpenPoolProject/stratum that referenced this pull request Nov 6, 2022
270: chore(deps): update actions/upload-artifact action to v3.1.1 r=renovate[bot] a=renovate[bot]

[![Mend Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com)

This PR contains the following updates:

| Package | Type | Update | Change |
|---|---|---|---|
| [actions/upload-artifact](https://togithub.com/actions/upload-artifact) | action | minor | `v3.0.0` -> `v3.1.1` |

---

### Release Notes

<details>
<summary>actions/upload-artifact</summary>

### [`v3.1.1`](https://togithub.com/actions/upload-artifact/releases/tag/v3.1.1)

[Compare Source](https://togithub.com/actions/upload-artifact/compare/v3.1.0...v3.1.1)

-   Update actions/core package to latest version to remove `set-output` deprecation warning [#&#8203;351](https://togithub.com/actions/upload-artifact/issues/351)

### [`v3.1.0`](https://togithub.com/actions/upload-artifact/releases/tag/v3.1.0)

[Compare Source](https://togithub.com/actions/upload-artifact/compare/v3.0.0...v3.1.0)

##### What's Changed

-   Bump [`@&#8203;actions/artifact](https://togithub.com/actions/artifact)` to v1.1.0 ([actions/upload-artifact#327)
    -   Adds checksum headers on artifact upload ([actions/toolkit#1095) ([actions/toolkit#1063)

</details>

---

### Configuration

📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

🚦 **Automerge**: Enabled.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box

---

This PR has been generated by [Mend Renovate](https://www.mend.io/free-developer-tools/renovate/). View repository job log [here](https://app.renovatebot.com/dashboard#github/OpenPoolProject/stratum).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNC4xOS4wIiwidXBkYXRlZEluVmVyIjoiMzQuMTkuMCJ9-->


271: chore(deps): update github/codeql-action action to v1.1.31 r=renovate[bot] a=renovate[bot]

[![Mend Renovate](https://app.renovatebot.com/images/banner.svg)](https://renovatebot.com)

This PR contains the following updates:

| Package | Type | Update | Change |
|---|---|---|---|
| [github/codeql-action](https://togithub.com/github/codeql-action) | action | minor | `v1.0.26` -> `v1.1.31` |

---

### Release Notes

<details>
<summary>github/codeql-action</summary>

### [`v1.1.31`](https://togithub.com/github/codeql-action/compare/v1.1.30...v1.1.31)

[Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.30...v1.1.31)

### [`v1.1.30`](https://togithub.com/github/codeql-action/compare/v1.1.29...v1.1.30)

[Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.29...v1.1.30)

### [`v1.1.29`](https://togithub.com/github/codeql-action/compare/v1.1.28...v1.1.29)

[Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.28...v1.1.29)

### [`v1.1.28`](https://togithub.com/github/codeql-action/compare/v1.1.27...v1.1.28)

[Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.27...v1.1.28)

### [`v1.1.27`](https://togithub.com/github/codeql-action/compare/v1.1.26...v1.1.27)

[Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.26...v1.1.27)

### [`v1.1.26`](https://togithub.com/github/codeql-action/compare/v1.1.25...v1.1.26)

[Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.25...v1.1.26)

### [`v1.1.25`](https://togithub.com/github/codeql-action/compare/v1.1.24...v1.1.25)

[Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.24...v1.1.25)

### [`v1.1.24`](https://togithub.com/github/codeql-action/compare/v1.1.23...v1.1.24)

[Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.23...v1.1.24)

### [`v1.1.23`](https://togithub.com/github/codeql-action/compare/v1.1.22...v1.1.23)

[Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.22...v1.1.23)

### [`v1.1.22`](https://togithub.com/github/codeql-action/compare/v1.1.21...v1.1.22)

[Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.21...v1.1.22)

### [`v1.1.21`](https://togithub.com/github/codeql-action/compare/v1.1.20...v1.1.21)

[Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.20...v1.1.21)

### [`v1.1.20`](https://togithub.com/github/codeql-action/compare/v1.1.19...v1.1.20)

[Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.19...v1.1.20)

### [`v1.1.19`](https://togithub.com/github/codeql-action/compare/v1.1.18...v1.1.19)

[Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.18...v1.1.19)

### [`v1.1.18`](https://togithub.com/github/codeql-action/compare/v1.1.17...v1.1.18)

[Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.17...v1.1.18)

### [`v1.1.17`](https://togithub.com/github/codeql-action/compare/v1.1.16...v1.1.17)

[Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.16...v1.1.17)

### [`v1.1.16`](https://togithub.com/github/codeql-action/compare/v1.1.15...v1.1.16)

[Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.15...v1.1.16)

### [`v1.1.15`](https://togithub.com/github/codeql-action/compare/v1.1.14...v1.1.15)

[Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.14...v1.1.15)

### [`v1.1.14`](https://togithub.com/github/codeql-action/compare/v1.1.13...v1.1.14)

[Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.13...v1.1.14)

### [`v1.1.13`](https://togithub.com/github/codeql-action/compare/v1.1.12...v1.1.13)

[Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.12...v1.1.13)

### [`v1.1.12`](https://togithub.com/github/codeql-action/compare/v1.1.11...v1.1.12)

[Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.11...v1.1.12)

### [`v1.1.11`](https://togithub.com/github/codeql-action/compare/v1.1.10...v1.1.11)

[Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.10...v1.1.11)

### [`v1.1.10`](https://togithub.com/github/codeql-action/compare/v1.1.9...v1.1.10)

[Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.9...v1.1.10)

### [`v1.1.9`](https://togithub.com/github/codeql-action/compare/v1.1.8...v1.1.9)

[Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.8...v1.1.9)

### [`v1.1.8`](https://togithub.com/github/codeql-action/compare/v1.1.7...v1.1.8)

[Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.7...v1.1.8)

### [`v1.1.7`](https://togithub.com/github/codeql-action/compare/v1.1.6...v1.1.7)

[Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.6...v1.1.7)

### [`v1.1.6`](https://togithub.com/github/codeql-action/compare/v1.1.5...v1.1.6)

[Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.5...v1.1.6)

### [`v1.1.5`](https://togithub.com/github/codeql-action/compare/v1.1.4...v1.1.5)

[Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.4...v1.1.5)

### [`v1.1.4`](https://togithub.com/github/codeql-action/compare/v1.1.3...v1.1.4)

[Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.3...v1.1.4)

### [`v1.1.3`](https://togithub.com/github/codeql-action/compare/v1.1.2...v1.1.3)

[Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.2...v1.1.3)

### [`v1.1.2`](https://togithub.com/github/codeql-action/compare/v1.1.1...v1.1.2)

[Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.1...v1.1.2)

### [`v1.1.1`](https://togithub.com/github/codeql-action/compare/v1.1.0...v1.1.1)

[Compare Source](https://togithub.com/github/codeql-action/compare/v1.1.0...v1.1.1)

### [`v1.1.0`](https://togithub.com/github/codeql-action/compare/v1.0.32...v1.1.0)

[Compare Source](https://togithub.com/github/codeql-action/compare/v1.0.32...v1.1.0)

### [`v1.0.32`](https://togithub.com/github/codeql-action/compare/v1.0.31...v1.0.32)

[Compare Source](https://togithub.com/github/codeql-action/compare/v1.0.31...v1.0.32)

### [`v1.0.31`](https://togithub.com/github/codeql-action/compare/v1.0.30...v1.0.31)

[Compare Source](https://togithub.com/github/codeql-action/compare/v1.0.30...v1.0.31)

### [`v1.0.30`](https://togithub.com/github/codeql-action/compare/v1.0.29...v1.0.30)

[Compare Source](https://togithub.com/github/codeql-action/compare/v1.0.29...v1.0.30)

### [`v1.0.29`](https://togithub.com/github/codeql-action/compare/v1.0.28...v1.0.29)

[Compare Source](https://togithub.com/github/codeql-action/compare/v1.0.28...v1.0.29)

### [`v1.0.28`](https://togithub.com/github/codeql-action/compare/v1.0.27...v1.0.28)

[Compare Source](https://togithub.com/github/codeql-action/compare/v1.0.27...v1.0.28)

### [`v1.0.27`](https://togithub.com/github/codeql-action/compare/v1.0.26...v1.0.27)

[Compare Source](https://togithub.com/github/codeql-action/compare/v1.0.26...v1.0.27)

</details>

---

### Configuration

📅 **Schedule**: Branch creation - At any time (no schedule defined), Automerge - At any time (no schedule defined).

🚦 **Automerge**: Enabled.

♻ **Rebasing**: Whenever PR becomes conflicted, or you tick the rebase/retry checkbox.

🔕 **Ignore**: Close this PR and you won't be reminded about this update again.

---

 - [ ] <!-- rebase-check -->If you want to rebase/retry this PR, check this box

---

This PR has been generated by [Mend Renovate](https://www.mend.io/free-developer-tools/renovate/). View repository job log [here](https://app.renovatebot.com/dashboard#github/OpenPoolProject/stratum).
<!--renovate-debug:eyJjcmVhdGVkSW5WZXIiOiIzNC4xOS4wIiwidXBkYXRlZEluVmVyIjoiMzQuMTkuMCJ9-->


Co-authored-by: renovate[bot] <29139614+renovate[bot]@users.noreply.github.com>
@wesdgray
Copy link

I ran into this problem with a service that handled large file uploads and the root cause was that our continuous deployment was killing our containers after a shorter than expected timeout. In our case, during deployment, hosts are taken out of the pool and in-flight requests are allowed to finish but it was not known to us at the time that a timeout was even in effect.

Also if you're wondering how I even stumbled upon this: https://youtu.be/9qljpi5jiMQ?t=722

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

4 participants