This repository has been archived by the owner on Feb 26, 2024. It is now read-only.
Create a web3 http provider that can handle massive JSON blobs without crashing due to string or buffer length issues #4381
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
davidmurdoch
commented
Oct 29, 2021
| result.structLogs.forEach((log: Buffer) => { | ||
| assert.strictEqual(log.length, SIZE); | ||
| }); | ||
| }) as any).timeout(0); |
There was a problem hiding this comment.
I don't know what is up with the mocha types, but it thinks it() returns void but it doesn't.
eggplantzzz
reviewed
Nov 1, 2021
| * @param {Object} payload | ||
| * @param {Function} callback triggered on end with (err, result) | ||
| */ | ||
| send( |
There was a problem hiding this comment.
Should we make this guy also implement request?
There was a problem hiding this comment.
No. This is a drop-in replacement for Web3HttpProvider, which doesn't implement request.
davidmurdoch
force-pushed
the
fix/streaming-web3-http-provider
branch
from
ae453d4
to
80339a1
Compare
davidmurdoch
commented
Mar 14, 2023
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Draft PR because I don't know the intricacies of Truffle well enough to be confident this works well enough. I'm not sure I replaced the right Web3.providers.HttpProvider calls.
This only supports http and https providers. It does NOT support ws(s) URLs. the default in truffle is
http, so it may still be valuable to ship without a WebSocket version. I imagine WebSockets support will be a bit more complicated to patch in.There is also only a single test added, and it's a strange one. The test mocks an RPC provider and just always returns a few gigs of data in the response. The test will fail with the infamous
Invalid String Lengtherror if themethodin the RPC request isn't "debug_traceTransaction" (as expected). So it does seem to work. The test takes 15-20s to run on my machine because of the massive amount of data that needs to be parsed, which isn't ideal #helpwanted :-)I've also smoke tested it with mainnet transaction
0x8bb8dc5c7c830bac85fa48acad2505e9300a91c3ff239c9517d0cae33b595090(which generates a 1.4 GB trace) and it works.I exported the class as
default, but I don't have a good reason. It really should be exported as a named class.Note: a ganache alpha that supports these giant traces will be released shortly after trufflesuite/ganache#1339 is merged.