ValueError: {'code': -32700, 'message': 'parse error'} and Can't pickle local object 'construct_web3_formatting_middleware'

[fridary]

What happened?

Erigon: 2.59.3-088fd8ef (last March 2024)

I have a problem to parallel querying local Erigon node. By the end of execution there is an error ValueError: {'code': -32700, 'message': 'parse error'}.
Error is always returned on last request, whether I query 4 or 50 transactions in Pool.
Although, if I test code on public node (https://eth-pokt.nodies.app in example) instead of 'http://127.0.0.1:8545', problem is gone.
Yes, perhaps problem is in Erigon. But if I do requests.post(), there is no error.
If I do another query like w3.eth.get_block(block), error is the same.
Any ideas what's wrong?
Update: it seems problem is in declaring w3 outside pool process together with inside (calculate_transaction()) for local nodes.

Erigon:
/home/fridary/erigon/build/bin/erigon --internalcl --datadir=/disk_sde/erigon --http.api=eth,erigon,engine,web3,net,debug,trace,txpool --authrpc.jwtsecret=/home/fridary/erigon/jwtsecret --metrics --prune.h.before=13916166 --prune.r.before=13916166 --prune.t.before=13916166 --prune.c.before=13916166 --torrent.download.rate=128mb

Code that produced the error

import requests
from web3 import Web3
from multiprocessing import Pool, cpu_count


# testnet = 'https://eth-pokt.nodies.app'
testnet = 'http://127.0.0.1:8545' # here is my local Erigon node


# def calculate_transaction(w3, i, hash_): # this gives error Can't pickle local object 'construct_web3_formatting_middleware'
def calculate_transaction(i, hash_):
    global w3

    # gives error
    transaction = w3.eth.get_transaction(hash_)

    # no error
    # transaction = requests.post(testnet, json={"method":"eth_getTransactionByHash","params":[hash_.hex()],"id":i,"jsonrpc":"2.0"}, headers={"Content-Type": "application/json"}).json()['result']

    print(i, transaction['hash'].hex())



w3 = Web3(Web3.HTTPProvider(testnet))
if not w3.is_connected():
    exit("w3 not connected")

block = w3.eth.get_block('latest')
transactions = block['transactions'][:min(cpu_count(), 4)]
    
params = []
for i, hash_ in enumerate(transactions):
    params.append((i, hash_))

pool = Pool(min(cpu_count(), 4))
pool.starmap_async(calculate_transaction, params).get()
pool.close()

Full error output

2 0x05aee6ea3d6a19b76e810e481736b16487a252525efc1640289b97e29eaafe08
3 0xbfc4114da15e7944038571362c191859f9a75655ed7a265fa9623102823851df
0 0x9a2b77146b8ab62fe9b1d50ca353f5faf841d744029ea913f76a702c79ec648c
multiprocessing.pool.RemoteTraceback:
"""
Traceback (most recent call last):
  File "/home/fridary/miniconda3/envs/eth/lib/python3.11/multiprocessing/pool.py", line 125, in worker
    result = (True, func(*args, **kwds))
                    ^^^^^^^^^^^^^^^^^^^
  File "/home/fridary/miniconda3/envs/eth/lib/python3.11/multiprocessing/pool.py", line 51, in starmapstar
    return list(itertools.starmap(args[0], args[1]))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fridary/python/ethereum/parallel_3.py", line 16, in calculate_transaction
    transaction = w3.eth.get_transaction(hash_)
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fridary/miniconda3/envs/eth/lib/python3.11/site-packages/web3/eth/eth.py", line 325, in get_transaction
    return self._get_transaction(transaction_hash)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fridary/miniconda3/envs/eth/lib/python3.11/site-packages/web3/module.py", line 75, in caller
    result = w3.manager.request_blocking(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fridary/miniconda3/envs/eth/lib/python3.11/site-packages/web3/manager.py", line 329, in request_blocking
    return self.formatted_response(
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fridary/miniconda3/envs/eth/lib/python3.11/site-packages/web3/manager.py", line 292, in formatted_response
    raise ValueError(error)
ValueError: {'code': -32700, 'message': 'parse error'}
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/fridary/python/ethereum/parallel_3.py", line 33, in <module>
    pool.starmap_async(calculate_transaction, params).get()
  File "/home/fridary/miniconda3/envs/eth/lib/python3.11/multiprocessing/pool.py", line 774, in get
    raise self._value
ValueError: {'code': -32700, 'message': 'parse error'}

### Fill this section in if you know how this could or should be fixed

_No response_

### web3 Version

6.16.0

### Python Version

3.11.8

### Operating System

Ubuntu 20.04.6 LTS

### Output from `pip freeze`

```shell
aiohttp==3.9.3
aiosignal==1.3.1
alchemy-sdk-py==0.2.0
attributedict==0.3.0
attrs==23.2.0
bitarray==2.9.2
blessings==1.7
cachetools==5.3.3
certifi==2024.2.2
chardet==5.2.0
charset-normalizer==3.3.2
codecov==2.1.13
colorama==0.4.6
coloredlogs==15.0.1
colour-runner==0.1.1
coverage==7.4.4
cytoolz==0.12.3
deepdiff==6.7.1
distlib==0.3.8
eth-abi==4.2.1
eth-account==0.11.0
eth-hash==0.7.0
eth-keyfile==0.8.0
eth-keys==0.5.0
eth-rlp==1.0.1
eth-typing==4.0.0
eth-utils==4.0.0
filelock==3.13.3
frozendict==2.3.10
frozenlist==1.4.1
hexbytes==0.3.1
humanfriendly==10.0
idna==3.6
inspecta==0.1.3
jsonschema==4.21.1
jsonschema-specifications==2023.12.1
lru-dict==1.2.0
markdown-it-py==3.0.0
mdurl==0.1.2
moralis==0.1.45
multidict==6.0.5
numpy==1.26.4
ordered-set==4.1.0
packaging==24.0
pandas==2.2.1
parsimonious==0.9.0
platformdirs==4.2.0
pluggy==1.4.0
protobuf==5.26.1
pycryptodome==3.20.0
Pygments==2.17.2
pyproject-api==1.6.1
python-dateutil==2.8.2
python-dotenv==1.0.1
pytz==2024.1
pyunormalize==15.1.0
referencing==0.34.0
regex==2023.12.25
requests==2.31.0
rich==13.7.1
rlp==4.0.0
rootpath==0.1.1
rpds-py==0.18.0
six==1.16.0
tabulate==0.9.0
termcolor==2.4.0
toolz==0.12.1
tox==4.14.2
typing_extensions==4.3.0
tzdata==2024.1
urllib3==1.26.18
virtualenv==20.25.1
web3==6.16.0
web3-input-decoder==0.1.11
websockets==12.0
yarl==1.9.4

[fridary]

Update: I figured out that if I don't declare w3 outside the calculate_transaction() function, I mean if I remove this code:

w3 = Web3(Web3.HTTPProvider(testnet))
if not w3.is_connected():
    exit("w3 not connected")

and if I declare w3 each time/process in calculate_transaction(), error will gone. So the problem is in defining w3 in/outside pool function. But how to fix I don't know, because in real program I have to define w3 also outside the pool processing.
The only reason could be to define w3 once at start and send w3 in func params like def calculate_transaction(w3, i, hash_), but then I will get this error:

Traceback (most recent call last):
  File "/home/fridary/python/ethereum/parallel_4.py", line 53, in <module>
    pool.starmap_async(calculate_transaction, params).get()
  File "/home/fridary/miniconda3/envs/eth/lib/python3.11/multiprocessing/pool.py", line 774, in get
    raise self._value
  File "/home/fridary/miniconda3/envs/eth/lib/python3.11/multiprocessing/pool.py", line 540, in _handle_tasks
    put(task)
  File "/home/fridary/miniconda3/envs/eth/lib/python3.11/multiprocessing/connection.py", line 206, in send
    self._send_bytes(_ForkingPickler.dumps(obj))
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/fridary/miniconda3/envs/eth/lib/python3.11/multiprocessing/reduction.py", line 51, in dumps
    cls(buf, protocol).dump(obj)
AttributeError: Can't pickle local object 'construct_web3_formatting_middleware.<locals>.formatter_middleware'

Maybe there is any way to close connection w3?

[fridary]

Update: this solution works, but it's ugly option. Creating new w3 every step on another address. Any other ideas?

testnet = 'http://127.0.0.1:8545/'
def calculate_transaction(i):
    w3 = Web3(Web3.HTTPProvider(testnet + f'?{i}'))

[kclowes]

On first glance, it looks like something to do with how we generate the cache key and multiprocessing. I'm glad you found a workaround though. We'll put this in our queue to look into. Thanks for the issue!