Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

memory usage increase significantly caused by eth.contract() #3139

Closed
mwrrlvkixj opened this issue Nov 2, 2023 · 1 comment
Closed

memory usage increase significantly caused by eth.contract() #3139

mwrrlvkixj opened this issue Nov 2, 2023 · 1 comment

Comments

@mwrrlvkixj
Copy link

mwrrlvkixj commented Nov 2, 2023

  • Version: 6.10.0
  • Python: 3.9
  • OS: osx
  • pip freeze output
aiohttp==3.8.5
aiosignal==1.3.1
async-timeout==4.0.2
attrs==23.1.0
base58==2.1.1
bech32==1.2.0
bitarray==2.8.0
boto3==1.21.19
botocore==1.24.46
cachetools==4.2.4
certifi==2023.7.22
charset-normalizer==2.0.12
contourpy==1.1.1
cycler==0.12.1
cytoolz==0.12.2
datadog==0.46.0
eth-abi==4.2.1
eth-account==0.9.0
eth-hash==0.5.2
eth-keyfile==0.6.1
eth-keys==0.4.0
eth-rlp==0.3.0
eth-typing==3.5.1
eth-utils==2.2.1
fonttools==4.43.1
frozenlist==1.4.0
guppy3==3.1.4.post1
hexbytes==0.2.2
idna==3.4
importlib-resources==6.1.0
ipfshttpclient==0.8.0a2
Jinja2==3.1.2
jmespath==0.10.0
jsonschema==4.18.4
jsonschema-specifications==2023.7.1
kiwisolver==1.4.5
lru-dict==1.2.0
markdown-it-py==3.0.0
MarkupSafe==2.1.3
matplotlib==3.8.0
mdurl==0.1.2
memory-profiler==0.61.0
memray==1.10.0
more-itertools==9.1.0
multiaddr==0.0.9
multidict==6.0.4
netaddr==0.8.0
numpy==1.22.0
packaging==23.1
pandas==1.5.3
parsimonious==0.9.0
Pillow==10.1.0
pluggy==0.13.1
protobuf==4.24.4
psutil==5.9.6
psycopg2==2.9.6
psycopg2-binary==2.9.5
py==1.11.0
pycryptodome==3.18.0
Pygments==2.16.1
pyparsing==3.1.1
pytest==5.4.1
python-dateutil==2.8.2
pytz==2021.3
pyunormalize==15.0.0
referencing==0.30.0
regex==2023.10.3
requests==2.27.1
rich==13.6.0
rlp==3.0.0
rpds-py==0.9.2
s3transfer==0.5.2
six==1.16.0
SQLAlchemy==1.4.47
SQLAlchemy-Utils==0.41.1
toolz==0.12.0
typing_extensions==4.7.1
urllib3==1.26.16
varint==1.0.2
wcwidth==0.2.6
web3==6.10.0
websockets==12.0
yarl==1.9.2
zipp==3.17.0

What was wrong?

Calling eth.contract() frequently and constantly causes memory usage increase a lot.

  def contract(self, contract, abi):
      start = time.time()
      try:
          self.api_count_tracking[CONTRACT_CALL] += 1
          c = self.w3_client.eth.contract(safe_to_check_sum_address(contract), abi=abi)
          end = time.time()
          self.time_tracking[CONTRACT_CALL] += end - start
          return c
      except Exception as e:
          logging.error(f'contract not found: contract_address is {contract} error is {e}. abi: {abi}')
          end = time.time()
          self.time_tracking[CONTRACT_CALL] += end - start

We call this function quite frequently, over 100 times/sec using multithreading and multiprocesses.
I used memory profiler to analyze it and it shows that ContractFactory = ContractFactoryClass.factory(self.w3, **kwargs) was allocated a lot of memory. Another candidate could be venv/lib/python3.9/site-packages/web3/_utils/datatypes.py:61

I found one issue mentioning this problem here: #2296

How can it be fixed?

Not sure how to fix that.


Note: We prefer to use issues to track our work. If you think you've encountered a bug in web3py or
have a feature request, you're in the right place. If you have implementation or usage questions,
please refer to our documentation and/or join the conversation
on discord.

@pacrob
Copy link
Contributor

pacrob commented May 20, 2024

Being tracked in #2296

@pacrob pacrob closed this as completed May 20, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants