Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add flag to ignore unknown EIP-2718 types when forking #2995

Closed
gzeoneth opened this issue Jul 26, 2022 · 54 comments · Fixed by #3302
Closed

Add flag to ignore unknown EIP-2718 types when forking #2995

gzeoneth opened this issue Jul 26, 2022 · 54 comments · Fixed by #3302
Assignees

Comments

@gzeoneth
Copy link

gzeoneth commented Jul 26, 2022

Note to reader: hardhat published a beta to fix this #2995 (comment)

Certain EVM blockchain (e.g. Arbitrum Nitro) have non-standard EIP-2718 typed transaction that would prevent hardhat fork from working due to Error HH604: Error running JSON-RPC server: Unknown transaction type 106

While it is rarely a good idea to use hardhat fork with these blockchain due to chain specific behavior and precompiles as discussed in #1902, some devs found it helpful to test non-chain specific logic with existing states using hardhat fork. To workaround the Unknown transaction type error, I am proposing to add an optional --fork-ignore-unknown-tx-type flag to skip those transactions.

You can reproduce the error with

hardhat node --fork https://goerli-rollup.arbitrum.io/rpc
	ArbitrumDepositTxType         = 100
	ArbitrumUnsignedTxType        = 101
	ArbitrumContractTxType        = 102
	ArbitrumRetryTxType           = 104
	ArbitrumSubmitRetryableTxType = 105
	ArbitrumInternalTxType        = 106
	ArbitrumLegacyTxType          = 120

Here is a draft implementation: https://github.com/gzeoneth/hardhat/tree/2.10.2-gzeon

$ git clone --branch 2.10.2-gzeon https://github.com/gzeoneth/hardhat.git
$ cd hardhat
$ yarn install
$ yarn build
$ touch hardhat.config.js
$ yarn hardhat node --fork https://rinkeby.arbitrum.io/rpc --fork-ignore-unknown-tx-type true

Also published on npm and can be installed with yarn alias

$ yarn add -D hardhat@npm:@gzeoneth/hardhat@2.10.2-gzeon-e972f1dda
$ yarn hardhat node --fork https://rinkeby.arbitrum.io/rpc --fork-ignore-unknown-tx-type true

27/8/2022:
I have upload a patch file here if you don't want to install my package, this is also compatible with a few more hardhat version e.g. 2.9.9, 2.10.2
https://gist.github.com/gzeoneth/99ac30932bf07f91661d7bc621ba1f50

You can install it using patch-package, just follow the step there and put hardhat+2.10.2.patch into the patches folder and run yarn install again. Please note that I did not update the source mapping in this patch file to improve compatibility but it might cause some issue debugging.

10/9/2022:
Fixed a few bug with 2.10.2-gzeon-e972f1dda, updated link above

24/10/2022: #2995 (comment)
A beta version of Hardhat is published that should fix this issue.

npm install hardhat@unknown-txs

(If you are using npm 7 or later, you might get some warnings about unsatisfied peer dependencies, but you can ignore those.)

@github-actions
Copy link
Contributor

This issue is also being tracked on Linear.

We use Linear to manage our development process, but we keep the conversations on Github.

LINEAR-ID: 63ce06f8-76ab-40ec-8fdc-3fda8f581808

@gzeoneth
Copy link
Author

gzeoneth commented Aug 3, 2022

Hey @feuGeneA any opinion on this? Can I move on to create a PR if this approach is ok?

@feuGeneA
Copy link
Contributor

feuGeneA commented Aug 8, 2022

Hi @gzeoneth . Thank you for the thoughtful and complete submission. I've just solicited more input from the team. We'll follow up soon.

@lzalvin
Copy link

lzalvin commented Aug 11, 2022

Would really like to be able to fork chains like arbitrum with the introduction of arbitrum nitro.

@feuGeneA
Copy link
Contributor

This issue has been added to our internal backlog. I can't say when we'll be able to prioritize it. But I can say that the more interest is shown here, the more likely we are to work on it sooner rather than later :)

@rafinskipg
Copy link

Hey, i assumed it was supported to fork arbitrum rinkeby for testing... I am surprised it does not work :( (i based my entire work on this working)

@gzeoneth
Copy link
Author

Hey, i assumed it was supported to fork arbitrum rinkeby for testing... I am surprised it does not work :( (i based my entire work on this working)

just fyi you can use this as a workaround for now

$ yarn add -D hardhat@npm:@gzeoneth/hardhat@2.10.1-gzeon-c8fe47dd4
$ yarn hardhat node --fork https://rinkeby.arbitrum.io/rpc --fork-ignore-unknown-tx-type true

@N00000b
Copy link

N00000b commented Aug 25, 2022

Yes I also need this. I am trying to test on a fork of Wanchain and am having the same issue. Thanks!

@lzalvin
Copy link

lzalvin commented Aug 27, 2022

looks great. works with hardhat_reset.

await hre.network.provider.request({
                method: 'hardhat_reset',
                params: [
                    {
                        forking: {
                            jsonRpcUrl: rpc,
                            enabled: true,
                            ignoreUnknownTxType: true,
                        },
                    },
                ],
            })

@gzeoneth
Copy link
Author

gzeoneth commented Aug 27, 2022

I have upload a patch file here if you don't want to install my package, this is also compatible with a few more hardhat version e.g. 2.9.9, 2.10.2
https://gist.github.com/gzeoneth/195cb16f00eaa32fa868e6c0192616c1 see op for more detail

@tom-s07
Copy link

tom-s07 commented Aug 31, 2022

This is absolutely needed, Arbitrum is starting to be one of the most active networks after Ethereum Mainnet. Thanks @gzeoneth for providing an interim solution, will give it a try!

@xcellsior
Copy link

xcellsior commented Aug 31, 2022

Definitely a needed change.

I am still getting the error:

Error HH604: Error running JSON-RPC server: Unknown transaction type 106, set --fork-ignore-unknown-tx-type true to ignore

after applying the patch.
I ran yarn hardhat node --fork https://rinkeby.arbitrum.io/rpc --fork-ignore-unknown-tx-type true

@ryandgoulding
Copy link

Patch works for me. Any update on this making it into an official release?

@0xyqrz
Copy link

0xyqrz commented Sep 2, 2022

This issue has been added to our internal backlog. I can't say when we'll be able to prioritize it. But I can say that the more interest is shown here, the more likely we are to work on it sooner rather than later :)

more interest is shown

@Zalexios
Copy link

Zalexios commented Sep 4, 2022

My infura and quicknodes Arbitrum mainnet RPC's stopped forking on Friday with this issue. I'm going to try the patch now, just showing interest @0xyqrz. Thanks everyone.

@Preston-Harrison
Copy link

Hey, i assumed it was supported to fork arbitrum rinkeby for testing... I am surprised it does not work :( (i based my entire work on this working)

just fyi you can use this as a workaround for now

$ yarn add -D hardhat@npm:@gzeoneth/hardhat@2.10.1-gzeon-c8fe47dd4
$ yarn hardhat node --fork https://rinkeby.arbitrum.io/rpc --fork-ignore-unknown-tx-type true

How can this be used with hardhat tests?

@gzeoneth
Copy link
Author

gzeoneth commented Sep 5, 2022

Hey, i assumed it was supported to fork arbitrum rinkeby for testing... I am surprised it does not work :( (i based my entire work on this working)

just fyi you can use this as a workaround for now

$ yarn add -D hardhat@npm:@gzeoneth/hardhat@2.10.1-gzeon-c8fe47dd4
$ yarn hardhat node --fork https://rinkeby.arbitrum.io/rpc --fork-ignore-unknown-tx-type true

How can this be used with hardhat tests?

see the first post you can use my patched hardhat or apply the patch file

@jacques-martin10
Copy link

@feuGeneA 👍 interest. thanks!

@0xchoripan
Copy link

The patch is not working for me when using hardhat_reset like this

  await network.provider.request({
    method: "hardhat_reset",
    params: [
      {
        forking: {
          jsonRpcUrl: config.networks.hardhat.forking.url,
          blockNumber,
          ignoreUnknownTxType: true,
        },
      },
    ],
  });

Before applying the patch

InternalError: Unknown transaction type 106

After applying the patch + adding ignoreUnknownTxType: true to hardhat_reset

InternalError: Unknown transaction type 106, set --fork-ignore-unknown-tx-type true to ignore

Using hardhat 2.10.1

@gzeoneth
Copy link
Author

gzeoneth commented Sep 6, 2022

  await network.provider.request({
    method: "hardhat_reset",
    params: [
      {
        forking: {
          jsonRpcUrl: "https://rinkeby.arbitrum.io/rpc",
          blockNumber: 15216420,
          ignoreUnknownTxType: true,
        },
      },
    ],
  });

working fine with me, do you have a sample script?

@AbigailTCameron
Copy link

  await network.provider.request({
    method: "hardhat_reset",
    params: [
      {
        forking: {
          jsonRpcUrl: "https://rinkeby.arbitrum.io/rpc",
          blockNumber: 15216420,
          ignoreUnknownTxType: true,
        },
      },
    ],
  });

where exactly are we supposed to add this aforementioned file if trying to run hardhat tests?

Since the Arbitrum nitro upgrade I have installed the patch, as well as followed the steps laid out above but keep getting an InternalError: Unknown transaction type 106, set --fork-ignore-unknown-tx-type true to ignore error.

@gzeoneth
Copy link
Author

gzeoneth commented Sep 9, 2022

  await network.provider.request({
    method: "hardhat_reset",
    params: [
      {
        forking: {
          jsonRpcUrl: "https://rinkeby.arbitrum.io/rpc",
          blockNumber: 15216420,
          ignoreUnknownTxType: true,
        },
      },
    ],
  });

where exactly are we supposed to add this aforementioned file if trying to run hardhat tests?

Since the Arbitrum nitro upgrade I have installed the patch, as well as followed the steps laid out above but keep getting an InternalError: Unknown transaction type 106, set --fork-ignore-unknown-tx-type true to ignore error.

I have no idea why it is not working, you can try this alternative patch where it always ignore unknown tx type.

@AbigailTCameron
Copy link

  await network.provider.request({
    method: "hardhat_reset",
    params: [
      {
        forking: {
          jsonRpcUrl: "https://rinkeby.arbitrum.io/rpc",
          blockNumber: 15216420,
          ignoreUnknownTxType: true,
        },
      },
    ],
  });

but which file are we adding the above stipulation to?

@gzeoneth
Copy link
Author

gzeoneth commented Sep 9, 2022

  await network.provider.request({
    method: "hardhat_reset",
    params: [
      {
        forking: {
          jsonRpcUrl: "https://rinkeby.arbitrum.io/rpc",
          blockNumber: 15216420,
          ignoreUnknownTxType: true,
        },
      },
    ],
  });

but which file are we adding the above stipulation to?

Fixed a few bug with 2.10.2-gzeon-e972f1dda, you can use this patch file
https://gist.github.com/gzeoneth/99ac30932bf07f91661d7bc621ba1f50

In hardhat.confg.ts you need something like

  networks: {
    hardhat: {
      forking: {
        url: "https://rinkeby.arbitrum.io/rpc",
        ignoreUnknownTxType: true
      }
    }
  },

Then it should work when using the hardhat network.

@zbirenbaum
Copy link

zbirenbaum commented Sep 28, 2022

None of the linked patches work when trying to set ignoreUnknownTxType: true, from within hardhat.config, and most of them don't seem to work for me even when passing the cli parameters as was the case for others here.

This issue has caused me no shortage of headaches and has been really frustrating for my team. I hope this gets resolved soon, but in the meantime, for anyone desperate to get this working and sick of patches breaking, after you apply one of the patches, inside of
node_modules/hardhat/internal/hardhat-network/provider/fork/ForkBlockchain.js

change:

class ForkBlockchain extends BlockchainBase_1.BlockchainBase {
    constructor(_jsonRpcClient, _forkBlockNumber, common, _forkIgnoreUnknownTxType) {
        super(common);
        this._jsonRpcClient = _jsonRpcClient;
        this._forkBlockNumber = _forkBlockNumber;
        this._forkIgnoreUnknownTxType = _forkIgnoreUnknownTxType;
        this._latestBlockNumber = this._forkBlockNumber;
    }

to:

class ForkBlockchain extends BlockchainBase_1.BlockchainBase {
    constructor(_jsonRpcClient, _forkBlockNumber, common, _forkIgnoreUnknownTxType) {
        super(common);
        this._jsonRpcClient = _jsonRpcClient;
        this._forkBlockNumber = _forkBlockNumber;
        this._forkIgnoreUnknownTxType = true;
        this._latestBlockNumber = this._forkBlockNumber;
    }

If you want to have a copy that is persistent across compilations so you can fork this repo and have it, just change node_modules/hardhat/src/internal/hardhat-network/provider/fork/ForkBlockchain.ts instead and recompile. Putting this._forkIgnoreUnknownTxType = true; will probably suffice.

To give some info that might help with a real solution: Its a bit of a pain to debug, but by the time node_modules/hardhat/internal/hardhat-network/provider/fork/ForkBlockchain.js is reachedthe network configuration has completely discarded the field. Printing the value of the network config at multiple points in the callstack shows that the field is discarded even after going in and manually setting it across numerous files within the callstack. I have no idea what possible reason there could be for this, but whoever wants to PR this definitely has their work cut out for them.

The fact that the docs don't mention that major chains like arbitrum can't be forked properly (on the contrary they make it sound like anything with an infura node works fine) is an absolute joke and wasted a whole week of my time before finding this issue. At the bare minimum there should be a clear error message, especially considering there have been reports of this going back over two whole months, and numerous issues have been closed as duplicates. I really hope they add a disclaimer somewhere so more devs don't waste their time on this.

@gzeoneth
Copy link
Author

@zbirenbaum mind to share your setup / repo that's not working with the above patches? Myself and a few others have been able to use it with hardhat.config but since I am not that familiar with hardhat I probably have missed something. It seems that hardhat will initialize the fork with the config but then do another hardhat_reset to change the blocknumber, but it won't have the ignore flag set, so I have to fix it by removing the ability to change the ignore flag using hardhat_reset gzeoneth@e972f1d, but it might have caused issue in other codepath.

Also @feuGeneA I think there are enough interest is shown here, when can we get a proper fix?

@zbirenbaum
Copy link

zbirenbaum commented Sep 28, 2022

@zbirenbaum mind to share your setup / repo that's not working with the above patches? Myself and a few others have been able to use it with hardhat.config but since I am not that familiar with hardhat I probably have missed something. It seems that hardhat will initialize the fork with the config but then do another hardhat_reset to change the blocknumber, but it won't have the ignore flag set, so I have to fix it by removing the ability to change the ignore flag using hardhat_reset gzeoneth@e972f1d, but it might have caused issue in other codepath.

Also @feuGeneA I think there are enough interest is shown here, when can we get a proper fix?

I'll make a proper patch file tomorrow and upload the repo, it's late here rn

@fvictorio
Copy link
Member

@zbirenbaum mind to share your setup / repo that's not working with the above patches? Myself and a few others have been able to use it with hardhat.config but since I am not that familiar with hardhat I probably have missed something. It seems that hardhat will initialize the fork with the config but then do another hardhat_reset to change the blocknumber, but it won't have the ignore flag set, so I have to fix it by removing the ability to change the ignore flag using hardhat_reset gzeoneth@e972f1d, but it might have caused issue in other codepath.

Also @feuGeneA I think there are enough interest is shown here, when can we get a proper fix?

We'll definitely prioritize this. Sorry everyone for not responding sooner!

Since a lot of you are struggling with this, maybe we can publish a dev version with some fix to be used in the meantime, until we can be confident we have a proper solution to release as a stable version. I'll let you all know when/if we do that.

@hugoroussel
Copy link

Any news on this? The patches do not work for me :(

@zbirenbaum
Copy link

With the 2.11.2 patch, and most of the others I believe, if you have

  networks: {
    hardhat: {
      forking: {
        url: "https://rinkeby.arbitrum.io/rpc",
        ignoreUnknownTxType: true
      }
    }
  },

defined in your hardhat config, npx hardhat node --fork-ignore-unknown-tx-type true will fail, but if it is not defined, the command will work. Regardless, any attempt to set the flag from your hardhat config will fail.

@QuiNoah00
Copy link

**ERROR** Failed to apply patch for package hardhat at path
  
    node_modules/hardhat

  This error was caused because hardhat has changed since you
  made the patch file for it. This introduced conflicts with your patch,
  just like a merge conflict in Git when separate incompatible changes are
  made to the same piece of code.

  Maybe this means your patch file is no longer necessary, in which case
  hooray! Just delete it!

  Otherwise, you need to generate a new patch file.

  To generate a new one, just repeat the steps you made to generate the first
  one.

  i.e. manually make the appropriate file changes, then run 

    patch-package hardhat

  Info:
    Patch file: patches/hardhat+2.10.2.patch
    Patch was made for version: 2.10.2
    Installed version: 2.11.2

patch-package finished with 1 error(s).

Getting this kind of error right now. What should I do?

{
    "name": "",
    "version": "1.0.0",
    "description": "# Deploy",
    "main": "index.js",
    "directories": {
        "test": "test"
    },
    "devDependencies": {
        "@nomicfoundation/hardhat-toolbox": "^1.0.2",
        "@openzeppelin/contracts": "^4.7.3",
        "hardhat": "npm:@gzeoneth/hardhat@^2.10.2-gzeon-e972f1dda"
    },
    "dependencies": {
        "dotenv": "^16.0.1",
        "hardhat-deploy": "^0.11.12",
        "patch-package": "^6.4.7"
    },
    "scripts": {
        "postinstall": "patch-package"
    }
}

This is my package.json file

@gzeoneth
Copy link
Author

gzeoneth commented Oct 3, 2022

@QuiNoah00 you don't need the patch if you are installing my npm package

@zbirenbaum
Copy link

zbirenbaum commented Oct 4, 2022

For anyone still struggling with this, I found a way to set up your project with the 2.11.2 patch here so that every time you run npx hardhat node it will automatically set up the network correctly for you, without having to worry about cli args or anything. You will need to create a .env file that contains the line
export INFURA_API_KEY='<your-api-key>' in the root of your project, or just change the jsonRpcUrl const to whatever rpc you use.

Just drop this in your hardhat.config, change it however you need to, and then run npx hardhat run --network localhost <your-script-here>

import '@nomicfoundation/hardhat-toolbox';
import { task } from "hardhat/config";
import { HardhatUserConfig } from 'hardhat/config';
import dotenv = require('dotenv');
dotenv.config();

import '@nomiclabs/hardhat-ethers'
import { TASK_NODE_SERVER_READY, } from "hardhat/builtin-tasks/task-names";
const INFURA_API_KEY = process.env.INFURA_API_KEY;
const jsonRpcUrl = `https://arbitrum-mainnet.infura.io/v3/${INFURA_API_KEY}`

const config: HardhatUserConfig = {
  defaultNetwork: "localhost",
  networks: {
    localhost: {
      forking: {
        url: jsonRpcUrl,
        enabled: true,
        ignoreUnknownTxType: true,
      }
    }
  },
  solidity: '0.8.9',
};

task(TASK_NODE_SERVER_READY).setAction(async (taskArgs, hre, runSuper) => {
  await hre.network.provider.request({
    method: 'hardhat_reset',
    params: [
      {
        forking: {
          jsonRpcUrl: jsonRpcUrl,
          enabled: true,
          ignoreUnknownTxType: true,
        },
      },
    ],
  })
  console.log('Set up forked network');
  runSuper(taskArgs);
});

export default config;

@b-pmcg
Copy link

b-pmcg commented Oct 20, 2022

any chance to get this working with ignoreUnknownTxType: true in the config?

@CodeLarkin
Copy link

This issue has been added to our internal backlog. I can't say when we'll be able to prioritize it. But I can say that the more interest is shown here, the more likely we are to work on it sooner rather than later :)

@feuGeneA Quite a bit of continued interest in this good ser!

@CodeLarkin
Copy link

CodeLarkin commented Oct 20, 2022

any chance to get this working with ignoreUnknownTxType: true in the config?

I couldn't get it to work with the 2.11.2 patch file provided so I modified it to use true in a bunch of places instead of that cli arg. Basically I hardcoded it to true. I am not sure how to create a github gist link like the others here, but here is my patch file: https://github.com/CodeLarkin/hardhat-patch-arbitrum/blob/main/hardhat%2B2.11.2.patch

My package.json deps include the following (and other stuff too ofc)

    "hardhat": "2.11.2",
    "patch-package": "^6.4.7",
    "postinstall-postinstall": "^2.1.0",

and then i have the following in the scripts section:

  "scripts": {
    "postinstall": "patch-package",
...

Edit: once you do this you don't need to use an extra cli arg or arg in hardhat.config.ts. Just run npx hardhat node or npx hardhat test and it should work with a forked arbitrum.

@vonhathanh
Copy link

vonhathanh commented Oct 21, 2022

I'm testing smart contract on arbitrum now, this feature is really needed to simulate test cases

@gzeoneth
Copy link
Author

I'm testing smart contract on arbitrum now, this feature is really needed to simulate test cases

Try to use the 2.10.2 patch in the original post.

@gzeoneth
Copy link
Author

@fvictorio updates?

@fvictorio
Copy link
Member

Hi everyone. We just published a beta version of Hardhat that should fix this issue. I would appreciate it if you can give it a try:

npm install hardhat@unknown-txs

(If you are using npm 7 or later, you might get some warnings about unsatisfied peer dependencies, but you can ignore those.)

If this seems to be working fine, we'll publish a stable version with this change as soon as possible.

Please try it out and let us know if it works (or doesn't!)

@b-pmcg
Copy link

b-pmcg commented Oct 25, 2022

ty @fvictorio! the beta is working for me, I'm able to fork the goerli arbitrum network successfully using the hardhat config. Will keep testing with this package and let you know if I run into issues.

@andy191x
Copy link

Nice job! The beta is working for me as well.

Installing the beta alone seems to hiccup on the hardhat-toolbox requirement, but I was able to workaround by installing the latest hardhat stable first, then hardhat-toolbox, then finally the beta. For example:

npm install hardhat
npx hardhat
# ... setup project with defaults
npm install hardhat@unknown-txs

@fvictorio
Copy link
Member

@andy191x yes, that's an unfortunate side effect of the way npm 7+ handles peer dependencies combined with how semver handles prerelease versions 😞

@sashaaldrick
Copy link

Hi everyone. We just published a beta version of Hardhat that should fix this issue. I would appreciate it if you can give it a try:

npm install hardhat@unknown-txs

(If you are using npm 7 or later, you might get some warnings about unsatisfied peer dependencies, but you can ignore those.)

If this seems to be working fine, we'll publish a stable version with this change as soon as possible.

Please try it out and let us know if it works (or doesn't!)

Worked perfectly for me! Forking now works with Arbitrum Goerli as usual. Thanks so much @fvictorio 👍

@fvictorio
Copy link
Member

This fix is now included in the latest release.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Jan 31, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.