Add a Command Line Interface #184

mraspaud · 2023-06-02T09:06:59Z

This PR adds a command line interface (CLI) to trollflow2.

This allows for easier launching in environments that might not have posttroll messaging enabled or for batch reprocessing.

codecov · 2023-06-02T09:13:16Z

Codecov Report

Attention: Patch coverage is 96.53465% with 7 lines in your changes are missing coverage. Please review.

Project coverage is 95.80%. Comparing base (146757d) to head (c4920a1).
Report is 6 commits behind head on main.

Files	Patch %	Lines
trollflow2/cli.py	91.48%	4 Missing ⚠️
trollflow2/tests/test_cli.py	97.77%	2 Missing ⚠️
trollflow2/launcher.py	97.22%	1 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff            @@
##             main     #184    +/-   ##
========================================
  Coverage   95.80%   95.80%            
========================================
  Files          13       15     +2     
  Lines        3000     3148   +148     
========================================
+ Hits         2874     3016   +142     
- Misses        126      132     +6

Flag	Coverage Δ
unittests	`95.80% <96.53%> (+<0.01%)`	⬆️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

gerritholl · 2023-06-02T09:35:33Z

This looks very interesting! Do you have an example of how to use this? To process old data, where we no longer have logfiles to recover a posstroll message from, we'd have to construct the JSON manually to pass this on the command line?

mraspaud · 2023-06-02T10:45:34Z

example usage:

satpy_cli -p pl.yaml ~/data/satellite/Meteosat-11/seviri/lvl1.5/2018/02/28/HRIT/* -m '{"start_time": "201802281500", "platform_name": "MSG4"}'

gerritholl · 2023-08-30T16:49:08Z

I tried

$ satpy_cli -p /home/gholl/checkouts/pytroll-dwd-config/geo-fci/trollflow2-fci-fdss-germany-classic.yaml *FDHSI*BODY---*0075_*.nc -m '{"start_time": "202306271220", "platform_name": "MTI1"}'

but this fails with

[ERROR: 2023-08-30 16:46:28 : trollflow2.launcher] Process crashed
Traceback (most recent call last):
  File "/opt/pytroll/pytroll_inst/mambaforge/envs/pytroll-py311/lib/python3.11/site-packages/trollflow2/launcher.py", line 396, in process_files
    process_jobs(config["workers"], jobs, produced_files)
  File "/opt/pytroll/pytroll_inst/mambaforge/envs/pytroll-py311/lib/python3.11/site-packages/trollflow2/launcher.py", line 440, in process_jobs
    cwrk.pop('fun')(job, **cwrk)
  File "/opt/pytroll/pytroll_inst/mambaforge/envs/pytroll-py311/lib/python3.11/site-packages/trollflow2/plugins/__init__.py", line 360, in save_datasets
    late_saver = save_dataset(scns, fmat, fmat_config, renames, compute=eager_writing)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/pytroll/pytroll_inst/mambaforge/envs/pytroll-py311/lib/python3.11/site-packages/trollflow2/plugins/__init__.py", line 254, in save_dataset
    with prepared_filename(fmat, renames) as filename:
  File "/opt/pytroll/pytroll_inst/mambaforge/envs/pytroll-py311/lib/python3.11/contextlib.py", line 137, in __enter__
    return next(self.gen)
           ^^^^^^^^^^^^^^
  File "/opt/pytroll/pytroll_inst/mambaforge/envs/pytroll-py311/lib/python3.11/site-packages/trollflow2/plugins/__init__.py", line 208, in prepared_filename
    directory, orig_filename = _prepare_filename_and_directory(fmat)
                               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/pytroll/pytroll_inst/mambaforge/envs/pytroll-py311/lib/python3.11/site-packages/trollflow2/plugins/__init__.py", line 182, in _prepare_filename_and_directory
    filename = os.path.join(directory, compose(fname_pattern, fmat))
                                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/pytroll/pytroll_inst/mambaforge/envs/pytroll-py311/lib/python3.11/site-packages/trollsift/parser.py", line 491, in compose
    return _strict_compose(fmt=fmt, keyvals=keyvals)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/pytroll/pytroll_inst/mambaforge/envs/pytroll-py311/lib/python3.11/site-packages/trollsift/parser.py", line 688, in _strict_compose
    return formatter.format(fmt, **keyvals)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/pytroll/pytroll_inst/mambaforge/envs/pytroll-py311/lib/python3.11/string.py", line 190, in format
    return self.vformat(format_string, args, kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/pytroll/pytroll_inst/mambaforge/envs/pytroll-py311/lib/python3.11/string.py", line 194, in vformat
    result, _ = self._vformat(format_string, args, kwargs, used_args, 2)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/pytroll/pytroll_inst/mambaforge/envs/pytroll-py311/lib/python3.11/string.py", line 247, in _vformat
    result.append(self.format_field(obj, format_spec))
                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/pytroll/pytroll_inst/mambaforge/envs/pytroll-py311/lib/python3.11/string.py", line 264, in format_field
    return format(value, format_spec)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
ValueError: Invalid format specifier '%Y%m%d_%H%M' for object of type 'str'

if I process the same files with

satpy_launcher.py -m /tmp/message /home/gholl/checkouts/pytroll-dwd-config/geo-fci/trollflow2-fci-fdss-germany-classic.yaml

then everything seems fine.

mraspaud · 2023-08-31T08:59:27Z

could you show us trollflow2-fci-fdss-germany-classic.yaml ?

gerritholl · 2023-08-31T10:07:41Z

product_list:
  output_dir:
    /data/pytroll/OUTBOXES/generic/
  reader:
    - fci_l1c_nc
  resampler: nearest
  areas:
    eurol:
      areaname: eurol
      products:
        'dwd_wv73':
          productname: 'dwd_wv73'
          formats:
            - writer: geotiff
              fname_pattern: "{platform_name}-{productname}-{areaname}-{start_time:%Y%m%d_%H%M}-{end_time:%H%M}-eager-devel-geotiff.tif"
              fill_value: 0

workers:
  - fun: !!python/name:trollflow2.plugins.create_scene
  - fun: !!python/name:trollflow2.plugins.load_composites
  - fun: !!python/name:trollflow2.plugins.resample
  - fun: !!python/name:trollflow2.plugins.save_datasets

And I know why it's happening too: normally start_time is a datetime, but by passing "202306271220" it's getting a string instead, which of course can't handle the format specification.

gerritholl · 2023-08-31T10:12:45Z

How would I encode a datetime object in JSON?

mraspaud · 2023-08-31T10:57:39Z

If you just use "{platform_name}-{productname}-{areaname}-{start_time}-{end_time}-eager-devel-geotiff.tif", it'll work...

mraspaud · 2023-08-31T11:31:01Z

Ok, so there is not default encoding for datetimes in json. However, in posttroll we do have a codec for datetimes that we could use here. I'll have a look.

For the cli tool recently added in pytroll#184, this adds an option to profile the run via dask and visualise the output as a bokeh plot.

mraspaud · 2023-12-14T12:08:01Z

@gerritholl @pnuu This is now ready I think

pnuu

Just one minor comment inline. LGTM

pnuu · 2024-03-13T13:49:26Z

trollflow2/cli.py

+    return log_config
+
+
+def datetime_decoder(dct):


Suggested change

def datetime_decoder(dct):

def datetime_decoder(datetimes):

I think the parameter needs a more descriptive name.

mraspaud added 3 commits June 2, 2023 09:25

Rename get_dask_client

a69d5de

Refactor launcher for easier use by the cli

978df29

Add the cli

a9d5439

mraspaud added the enhancement New feature or request label Jun 2, 2023

mraspaud self-assigned this Jun 2, 2023

mraspaud requested a review from pnuu as a code owner June 2, 2023 09:07

mraspaud added 2 commits June 2, 2023 11:31

Add entry point for the cli

2bfef9b

Fix produced files type

cd44eb6

mraspaud and others added 2 commits June 2, 2023 12:47

Remove stickler config

13f9285

Merge branch 'main' into feature-cli

a9b9810

gerritholl added a commit to gerritholl/trollflow2 that referenced this pull request Oct 24, 2023

Added option to run CLI with dask profiler

328642b

For the cli tool recently added in pytroll#184, this adds an option to profile the run via dask and visualise the output as a bokeh plot.

gerritholl mentioned this pull request Oct 24, 2023

Profile CLI run through dask #193

Open

4 tasks

mraspaud added 2 commits December 14, 2023 13:04

Allow parsing datetimes

96610b1

Merge branch 'main' into feature-cli

50239c7

gerritholl approved these changes Mar 13, 2024

View reviewed changes

Merge branch 'main' into feature-cli

c4920a1

pnuu approved these changes Mar 13, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add a Command Line Interface #184

Add a Command Line Interface #184

mraspaud commented Jun 2, 2023

codecov bot commented Jun 2, 2023 •

edited

gerritholl commented Jun 2, 2023

mraspaud commented Jun 2, 2023

gerritholl commented Aug 30, 2023

mraspaud commented Aug 31, 2023

gerritholl commented Aug 31, 2023

gerritholl commented Aug 31, 2023

mraspaud commented Aug 31, 2023

mraspaud commented Aug 31, 2023

mraspaud commented Dec 14, 2023

pnuu left a comment

pnuu Mar 13, 2024

Add a Command Line Interface #184

Are you sure you want to change the base?

Add a Command Line Interface #184

Conversation

mraspaud commented Jun 2, 2023

codecov bot commented Jun 2, 2023 • edited

Codecov Report

gerritholl commented Jun 2, 2023

mraspaud commented Jun 2, 2023

gerritholl commented Aug 30, 2023

mraspaud commented Aug 31, 2023

gerritholl commented Aug 31, 2023

gerritholl commented Aug 31, 2023

mraspaud commented Aug 31, 2023

mraspaud commented Aug 31, 2023

mraspaud commented Dec 14, 2023

pnuu left a comment

Choose a reason for hiding this comment

pnuu Mar 13, 2024

Choose a reason for hiding this comment

codecov bot commented Jun 2, 2023 •

edited