RuntimeError: Input/output error - Reading a small .nc file.

[msw09090]

NetCDF4 Details:
ca-certificates-2022.12.7
certifi-2022.12.7
cftime-1.5.1.1
conda-22.11.1
netcdf4-1.5.7
openssl-1.1.1s
pluggy-1.0.0
ruamel.yaml-0.17.21
ruamel.yaml.clib-0.2.6
Environment Information:
OS - rhel centos fedora version 8.6 (Linux)
Python - 3.9.12
Issue:

File "/source_code/utilities/MRMS/MRMS_animation.py", line 523, in
rdr = np.asarray(ncin['MergedReflectivityQCComposite_500mabovemeansealevel'][0,:,:])
File "src/netCDF4/_netCDF4.pyx", line 4967, in netCDF4._netCDF4.Variable.getitem
File "src/netCDF4/_netCDF4.pyx", line 5924, in netCDF4._netCDF4.Variable._get
File "src/netCDF4/_netCDF4.pyx", line 2028, in netCDF4._netCDF4._ensure_nc_success
RuntimeError: Input/output error
Error in code:
rdr = np.asarray(ncin['MergedReflectivityQCComposite_500mabovemeansealevel'][0,:,:])

I cannot replicate this problem with the same file... Usually it works, sometimes it doesn't with the same file (seems like a connection problem?). This problem appears intermittently and will go away if I just wait. It does reoccur though - I typically get the error if I read many files (~365 to 1000+ files) using one or more scripts.

[jswhit]

filesystem issue?

[msw09090]

If that's the case, would 'sleeping' the script for 1sec after closing each .nc file stabilize the netCDF4 module?

Something like this:
nc = netCDF4.Dataset(infile)
nc.close()
sleep(1)

[jswhit]

I can't explain it, but neither can I reproduce it. WIthout that, we don't have much to go on.

[jswhit]

Note that 'input/output error' is coming from the OS, and usually indicates a file system issue (such as the disk being full, or a hardware issue).

[msw09090]

Hello jswhit,

Thank you for your feedback!

I can't replicate the error with specific input files so I agree that there's not much I/we can do about this one. However, I have enough disk space to run the script. By process of elimination, this must be a file system issue.

[msw09090]

It looks like I was reading too many .nc files (365 times on a 7000x3000 grid) for my OS to handle. I think it was a memory issue (or something related). On top of reading 365 .nc files, I also pre-allocated a 365x7000x3000 integer matrix which probably exacerbated the problem.

It works well now that I binned my yearly .nc files (365x7000x3000) to monthly .nc files (~30x7000x3000).