Penman-Monteith forcing (and the struggles of ESMValTool)

[BSchilperoort]

In PR #393 I attempted to add a generic forcing that would provide users with the meteo data required for the Penman-Monteith equation. I.e., air temperature, net radiation, vapor pressure deficit, wind speed.

However, I encountered a few issues:

• Debugging is impossible with esmvaltool/esmvalcore as it locks up the threads. Which makes fixing problems in the code take a very long time as emsvaltool is slow.
• The CMIP data is limited in the different variables that are available
• The CMORized ERA5 data variables do not align well with the CMIP6 variables
  • wind speed consists of two components instead of mean horizontal speed
  • radiation has the net long/short wave and downwards long/short wave, instead of the 4 up/down long/short components
  • the dewpoint temperature is on a different mip table than all other variables. Which causes it to have different time bounds. Even though in the original ERA5 data from the CDS all data is on the same coordinates. Also, what is the difference between Eday and day because this is impossible to find in ESMValTool documentation, CMOR documentation, CMIP documentation.

Due to the issues piling up, we decided to put this on halt and finish up the PR without PM.

Here is how far I got:
https://gist.github.com/BSchilperoort/254fb3c5a54155b72e08861f40cb0e14

Of course this is a frustrating experience for myself, but I think this also touches on a more general problem: ESMValTool is built for evaluating climate model output. We're trying to use it for hydrological model input. This will clash in many ways, especially with ERA5 not fitting in CMIP properly.

While I do think that there is no alternative for ESMValTool when trying to get climate model data forcing data, I think that it might be better to give up on ERA5 + ESMValTool as it makes things a lot more complicated than it needs to be.

[BSchilperoort]

While I do think that there is no alternative for ESMValTool when trying to get climate model data forcing data, I think that it might be better to give up on ERA5 + ESMValTool as it makes things a lot more complicated than it needs to be.

For an "eWaterCycle v3.0" I would propose to make esmvalcore optional, and make it not load on import ewatercycle. A default forcing generator can instead use, e.g. ERA5.

In the future it might be possible to directly use xarray to get ecmwf data, see https://github.com/bopen/xarray-ecmwf. This could make for a much more lightweight default forcing generator.

[RolfHut]

good one to discuss in person Thursday, while I support stopping with trying to build a generic Penman-Monteith forcing, I do not support the notion of stopping with ESMValTool as integral part of eWaterCycle. Let's discuss in person and report back here on outcomes

[Peter9192]

I recognize many of your frustrations, and I could agree with making esmvalcore optional (but not abandoning it).

I think that it might be better to give up on ERA5 + ESMValTool as it makes things a lot more complicated than it needs to be.

Lot's of climate studies also include era5 as a reference. A properly cmorized version of era5 would be very welcome. Whether that should be our concern is up for debate.

[RolfHut]

I also recognize the struggles (didn't get a simple ERA5 CMIP comparison to run yesterday...)

I would like to postpone this choice after we know if big Horizon proposal just submitted gets accepted. If it does, we are building eWaterCycle on top of the DestinE Service Platform with it's own access to ESA data lake (which includes CMIP and ECMWF data). If we don't get that, we can continue this discussion on the V3 requirements

[BSchilperoort]

To clarify: I did not want to advocate for abandoning esmvaltool, but mostly making it

1. Not load in by default
2. Not the main way of making ERA5 forcing (if at some point we have time to work on this)

A properly cmorized version of era5 would be very welcome. Whether that should be our concern is up for debate.

This would be a pretty major project, as one of the main things that are lacking now is the small overlap between the MIP table variables and the ERA5 CMORized variables.
To solve this, a lot of variables would have to be derived or calculated from the base ERA5 output. This would require a major time investment, as well as an enormous amount of storage if we'd want to have access to this data.

[Peter9192]

as well as an enormous amount of storage if we'd want to have access to this data

Note that ESMValTool supports on-the-fly cmorization. CMORizers can be added per-variable, see https://github.com/ESMValGroup/ESMValCore/blob/main/esmvalcore/cmor/_fixes/native6/era5.py and https://docs.esmvaltool.org/en/latest/input.html#datasets-in-native-format

[BSchilperoort]

Additional issue encountered by Rolf here. ESMValTool uses the center of the time bounds (i.e. noon) for daily data.

This will conflict with other daily data loaded with e.g. pandas and xarray. It's also not really following the cf-conventions. According to those 2001-01-01 is interpreted as 2001-01-01T00:00:00.