v0.3.2

weather-mv is much faster now & equipped with Cloud-Optimized Geotiffs (GOGs) ingestion. weather-dl is enhanced to support MARS syntax in JSON config files and restriction for max-number of workers.

We're happy to welcome @deepgabani8 to the weather-tools dev team !

Current Status

weather-dl: Fixes and parser system improvements

• Fixed error while parsing new-line separated date-values.
• JSON config files now support MARS syntax.
• New syntax supported: now, users can specify MARS range syntax in reverse orders as well (e.g. 2020-01-01/to/2018-01-01/by/-1).
• Prevent exhaustion of quotas: Based on current approach for the downloader, we've capped the max number of workers to N i.e. possible simultaneous requests + fudge factor.

weather-mv: Performance improvements and support for COGs ingestion

• Substantial performance improvement !
• Added flag to control in-memory copying of dataset. By default the dataset is opened in-memory, the user can restrict it by passing the --disable_in_memory_copy flag.
• Added validation to alert the user earlier that the BigQuery table and temp location (cloud bucket) need to be in the same region. Users can skip this validation by passing the -s, --skip-region-validation flag.
• Added support for ingestion of COGs into BigQuery.
• Updated doc (README.md) of the tool to remove duplicate flags from sample examples.

General

• Fixed typo in contribution guide (CONTRIBUTING.md).

What's Changed

• Validates non-compatible regions scenarios in weather_mv tool. by @mahrsee1997 in #155
• Changes for #75, CLI-support for user controlled in-memory copy operation by @ksic8 in #154
• Candidate implementation to speed up row extraction. by @alxmrs in #146
• Fixed typo in CONTRIBUTING.md by @mahrsee1997 in #156
• Fix parsing newline separated date values by @mahrsee1997 in #161
• Support ingestion of COGs into BigQuery by @mahrsee1997 in #158
• DL: Add support for reverse order in MARS range syntax by @deepgabani8 in #162
• DL: Add support for MARS syntax in JSON config by @deepgabani8 in #163
• Removed duplicate flag from sample examples given in Readme.md of weather_mv tool. by @mahrsee1997 in #168
• Cap max number of workers in weather-dl. by @mahrsee1997 in #170

New Contributors

• @deepgabani8 made their first contribution in #162

Full Changelog: v0.3.1...v0.3.2