Request optimization #1147

fredrik-corneliusson · 2022-11-06T12:44:26Z

I noticed that getting earnings did a lot of duplicate requests that made it quite slow.
When looking into this I found it easier to have one place to retrieve data in order to ease caching and speed up operations and reduce code duplication. Needs Python 3.6
As I use lru_cache I needed to drop Python versions before 3.6, update package metadata to reflect this.

Merge all dev updates into main

Improve error handling

… operations and reduce code duplication. Needs Python 3.6

ValueRaider · 2022-11-06T16:02:34Z

What speedup are you seeing?

fredrik-corneliusson · 2022-11-06T18:05:04Z

See before and after picture.
A good deal, at least running against the current dev branch

Also many unnecessary requests was removed because of cleaner code.
from:

DEBUG:urllib3.connectionpool:https://finance.yahoo.com:443 "GET /quote/GOOGL/holders HTTP/1.1" 200 None
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): finance.yahoo.com:443
DEBUG:urllib3.connectionpool:https://finance.yahoo.com:443 "GET /quote/GOOGL HTTP/1.1" 200 None
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): query1.finance.yahoo.com:443
DEBUG:urllib3.connectionpool:https://query1.finance.yahoo.com:443 "GET /ws/fundamentals-timeseries/v1/finance/timeseries/GOOGL?symbol=GOOGL&type=trailingPegRatio&period1=1652028925&period2=1667843725 HTTP/1.1" 200 363
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): finance.yahoo.com:443
DEBUG:urllib3.connectionpool:https://finance.yahoo.com:443 "GET /quote/GOOGL/financials HTTP/1.1" 200 None
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): finance.yahoo.com:443
DEBUG:urllib3.connectionpool:https://finance.yahoo.com:443 "GET /quote/GOOGL/financials HTTP/1.1" 200 None
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): finance.yahoo.com:443
DEBUG:urllib3.connectionpool:https://finance.yahoo.com:443 "GET /quote/GOOGL/financials HTTP/1.1" 200 None
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): query2.finance.yahoo.com:443
DEBUG:urllib3.connectionpool:https://query2.finance.yahoo.com:443 "GET /ws/fundamentals-timeseries/v1/finance/timeseries/GOOGL?symbol=GOOGL&type=annualTotalRevenue,... HTTP/1.1" 200 None
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): finance.yahoo.com:443
DEBUG:urllib3.connectionpool:https://finance.yahoo.com:443 "GET /quote/GOOGL/financials HTTP/1.1" 200 None
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): query2.finance.yahoo.com:443
DEBUG:urllib3.connectionpool:https://query2.finance.yahoo.com:443 "GET /ws/fundamentals-timeseries/v1/finance/timeseries/GOOGL?symbol=GOOGL&type=quarterlyTotalRevenue,quarterlyOperatingRevenue,... HTTP/1.1" 200 None
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): finance.yahoo.com:443
DEBUG:urllib3.connectionpool:https://finance.yahoo.com:443 "GET /quote/GOOGL/balance-sheet HTTP/1.1" 200 None
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): finance.yahoo.com:443
DEBUG:urllib3.connectionpool:https://finance.yahoo.com:443 "GET /quote/GOOGL/balance-sheet HTTP/1.1" 200 None
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): query2.finance.yahoo.com:443
DEBUG:urllib3.connectionpool:https://query2.finance.yahoo.com:443 "GET /ws/fundamentals-timeseries/v1/finance/timeseries/GOOGL?symbol=GOOGL&type=annualTotalAssets,...0 HTTP/1.1" 200 None
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): finance.yahoo.com:443
DEBUG:urllib3.connectionpool:https://finance.yahoo.com:443 "GET /quote/GOOGL/balance-sheet HTTP/1.1" 200 None
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): query2.finance.yahoo.com:443
DEBUG:urllib3.connectionpool:https://query2.finance.yahoo.com:443 "GET /ws/fundamentals-timeseries/v1/finance/timeseries/GOOGL?symbol=GOOGL&type=quarterlyTotalAssets,... HTTP/1.1" 200 None
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): finance.yahoo.com:443
DEBUG:urllib3.connectionpool:https://finance.yahoo.com:443 "GET /quote/GOOGL/cash-flow HTTP/1.1" 200 None
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): finance.yahoo.com:443
DEBUG:urllib3.connectionpool:https://finance.yahoo.com:443 "GET /quote/GOOGL/cash-flow HTTP/1.1" 200 None
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): query2.finance.yahoo.com:443
DEBUG:urllib3.connectionpool:https://query2.finance.yahoo.com:443 "GET /ws/fundamentals-timeseries/v1/finance/timeseries/GOOGL?symbol=GOOGL&type=annualCashFlowsfromusedinOperatingActivitiesDirect,... HTTP/1.1" 200 None
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): finance.yahoo.com:443
DEBUG:urllib3.connectionpool:https://finance.yahoo.com:443 "GET /quote/GOOGL/cash-flow HTTP/1.1" 200 None
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): query2.finance.yahoo.com:443
DEBUG:urllib3.connectionpool:https://query2.finance.yahoo.com:443 "GET /ws/fundamentals-timeseries/v1/finance/timeseries/GOOGL?symbol=GOOGL&type=quarterlyCashFlowsfromusedinOperatingActivitiesDirect... HTTP/1.1" 200 None
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): finance.yahoo.com:443
DEBUG:urllib3.connectionpool:https://finance.yahoo.com:443 "GET /quote/GOOGL/analysis HTTP/1.1" 200 None
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): finance.yahoo.com:443
DEBUG:urllib3.connectionpool:https://finance.yahoo.com:443 "GET /quote/GOOGL/analysis HTTP/1.1" 200 None
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): finance.yahoo.com:443

To:

DEBUG:urllib3.connectionpool:https://finance.yahoo.com:443 "GET /quote/GOOGL/holders HTTP/1.1" 200 None
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): finance.yahoo.com:443
DEBUG:urllib3.connectionpool:https://finance.yahoo.com:443 "GET /quote/GOOGL HTTP/1.1" 200 None
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): query1.finance.yahoo.com:443
DEBUG:urllib3.connectionpool:https://query1.finance.yahoo.com:443 "GET /ws/fundamentals-timeseries/v1/finance/timeseries/GOOGL?symbol=GOOGL&type=trailingPegRatio&period1=1652028653&period2=1667843453 HTTP/1.1" 200 363
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): finance.yahoo.com:443
DEBUG:urllib3.connectionpool:https://finance.yahoo.com:443 "GET /quote/GOOGL/financials HTTP/1.1" 200 None
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): finance.yahoo.com:443
DEBUG:urllib3.connectionpool:https://finance.yahoo.com:443 "GET /quote/GOOGL/analysis HTTP/1.1" 200 None
DEBUG:urllib3.connectionpool:Starting new HTTPS connection (1): finance.yahoo.com:443

fredrik-corneliusson · 2022-11-06T18:35:02Z

Please note that it is specifically against the dev branch that has severe performance degradation for earnings (10s instead of under 3s in 0.1.85 in my tests), This patch cuts the time to get earnings to around 3.5s so still slower than 0.1.85. However the big win is that using this data-class will make performance bugs harder to introduce in the future.

ValueRaider · 2022-11-06T19:24:08Z

Great. My only comment is - is cache_maxsize set appropriately? I've not used lru_cache before so don't know if Ticker objects share same cache.

# Conflicts: # tests/ticker.py # yfinance/base.py

fredrik-corneliusson · 2022-11-06T23:23:00Z

I just set it to something I figured would be ok. I did not want it to be unlimited.
Ticker objects share the same cache:
https://docs.python.org/3/library/functools.html#functools.lru_cache
"If a method is cached, the self instance argument is included in the cache"

ValueRaider · 2022-11-07T00:36:59Z

I've done some benchmarking - a loop of tickers accessing info[] or cashflow. I measure:

11 MB per ticker
1.7 MB per get() cache element
3.6 MB per get_json_data_stores() cache element

1000 elements in get_json_data_stores() cache will hit 5.3 GB

I also see no difference with the freezeargs enabled/disabled - what is purpose?

Benchmark in case interested:

process = psutil.Process(os.getpid())
mem_baseline = process.memory_info().rss
for tkr in tkrs:
	dat = yf.Ticker(tkr, session=session)
	# dat.info
	try:
		dat.cashflow
	except TypeError:
		pass
mem_used = process.memory_info().rss - mem_baseline
# Note: only works with freezeargs disabled
cs = dat._data.get_json_data_stores.cache_info()
print("get_json_data_stores() cache stats:")
print(cs)
print("get_json_data_stores() cache MB per # = {}".format(round(mem_used/1e6/cs.currsize, 1)))

…n of memory usage. Also fixed warning about wrong type used for dataframe index.

fredrik-corneliusson · 2022-11-07T09:39:33Z

Good investigation, I decreased the cache size to 128 so it should now use less than 600 Mb,
Most of the use of the cache is probably close together in time so I do not think it will harm performance.
The freezeargs is needed if you pass headers (a mutable dict) as caching relies on the hash function of the arguments when it saves it to cache and looks it up.

…u_cached methods.

fredrik-corneliusson · 2022-11-08T00:39:01Z

I've lowered the cache size to 64 and fixed so cache_clear and cache_info should now work without disabling freezeargs.

ValueRaider · 2022-11-08T01:07:31Z

I'm happy to merge but think sensible to wait 24H in case you ruminate more changes.

fredrik-corneliusson · 2022-11-08T12:06:21Z

I think the PR is ok to merge.
Any more changes will be in separate PR.
Next big optimization to look into would be lazy fetching of the the different scraping endpoints. Part of why it is slow not is the for many properties all financial is fetched even if only parts of it is requested. But that would be a too big change for this PR.

ValueRaider and others added 10 commits October 25, 2022 18:13

Merge pull request ranaroussi#1117 from ranaroussi/dev

bec5b38

Merge all dev updates into main

Merge pull request ranaroussi#1119 from ranaroussi/dev

3f33aa0

Improve error handling

Bump version to 0.2.0rc1 - big update

fb5c67b

Have one place to retrieve data in order to ease caching and speed up…

05520ee

… operations and reduce code duplication. Needs Python 3.6

Dropped Python versions before 3.6 from package support metadata

07e19f1

Revert to require a pandas version that supports 3.6.1

9f9f7b0

Removed unused get_html method.

743f3ac

Removed extra requests logging used for debugging.

2cc3cbb

Fixed regression in PR and cleaned up .gitignore

157b452

Cleaned up .gitignore

438f512

Improved TestTickerHolders test.

2fed55a

Fixed regression bug with balance_sheet and added test for it.

2be7187

fredrik-corneliusson added 4 commits November 6, 2022 20:49

Added more tests for Ticker and missing dependencies.

112fd5c

Merge branch 'ranaroussi:main' into request_optimization

f6c3118

Merge branch 'dev' into request_optimization

f96dfc2

# Conflicts: # tests/ticker.py # yfinance/base.py

Tried to clean up the temp folder creation in test.

a775669

Decreased default cache_maxsize for lru_cache after some investigatio…

5bfbec5

…n of memory usage. Also fixed warning about wrong type used for dataframe index.

Lowered lru_cache size and made cache_info and cache_clear work on lr…

c7cf437

…u_cached methods.

ValueRaider merged commit 23e8423 into ranaroussi:dev Nov 8, 2022

ValueRaider mentioned this pull request Nov 11, 2022

Merge dev -> main for release 0.2.0rc2 #1168

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Request optimization #1147

Request optimization #1147

fredrik-corneliusson commented Nov 6, 2022

ValueRaider commented Nov 6, 2022

fredrik-corneliusson commented Nov 6, 2022 •

edited

fredrik-corneliusson commented Nov 6, 2022

ValueRaider commented Nov 6, 2022 •

edited

fredrik-corneliusson commented Nov 6, 2022

ValueRaider commented Nov 7, 2022

fredrik-corneliusson commented Nov 7, 2022

fredrik-corneliusson commented Nov 8, 2022

ValueRaider commented Nov 8, 2022

fredrik-corneliusson commented Nov 8, 2022

Request optimization #1147

Request optimization #1147

Conversation

fredrik-corneliusson commented Nov 6, 2022

ValueRaider commented Nov 6, 2022

fredrik-corneliusson commented Nov 6, 2022 • edited

fredrik-corneliusson commented Nov 6, 2022

ValueRaider commented Nov 6, 2022 • edited

fredrik-corneliusson commented Nov 6, 2022

ValueRaider commented Nov 7, 2022

fredrik-corneliusson commented Nov 7, 2022

fredrik-corneliusson commented Nov 8, 2022

ValueRaider commented Nov 8, 2022

fredrik-corneliusson commented Nov 8, 2022

fredrik-corneliusson commented Nov 6, 2022 •

edited

ValueRaider commented Nov 6, 2022 •

edited