-
Notifications
You must be signed in to change notification settings - Fork 285
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Too much memory usage for composite processing #2764
Comments
First thing: do all the loading in the first Things to try:
|
More details on performance frequently asked questions: https://satpy.readthedocs.io/en/stable/faq.html I agree with everything Panu said, but additionally want to point out that if your destination/target area definition for resampling is in the satellite's native projection then there are other options besides resampling with nearest neighbor or gradient search that would likely be faster. Otherwise, how does your example script compare with what you are actually doing? You have two |
Describe the bug
for creating composite products (even when I ignore atmospheric correction) out of ABI imagery, peak memory usage exceeds 30 GB. I suspect something may be going wrong, as it also takes over 8 minutes. Are there any best practices for increasing the speed? For example, should we tweak parameters such as chunk size to find the optimum? Additionally, can we implement caching or pre-compute certain data (as the field of view of ABI is fixed) to increase the speed for subsequent runs?
To Reproduce
scn.load(scn.available_dataset_names())
scn_resmp = scn.resample(destination=dst_area_def, radius_of_influence=50000)
composite = 'true_color_raw'
scn_resmp.load([composite])
dataset = scn_resmp[composite]
plt.figure()
img = get_enhanced_image(dataset)
img_data = img.data
img_data.plot.imshow(vmin=0, vmax=1, rgb='bands')
img_data.plot.imshow(rgb='bands')
Expected behavior
As the file size are much less and using dask, I expected it be executed much smoother(in usual 8 or 16 Gb ram system) and faster(in less than 2-3 mins).
Actual results
During visualization, I encounter too many of these warnings. I'm unsure of how much they are related to the performance issue.
lib\site-packages\dask\core.py:119: RuntimeWarning: invalid value encountered in cos return func(*(_execute_task(a, cache) for a in args))
Environment Info:
The text was updated successfully, but these errors were encountered: