Job output transform #548
Job output transform #548
Conversation
Direct Links in job output + makefile + tests
Testing transformer engine added
Remove requirements-trfm Add to main requirements
Why not pandas
Update requirements
Ajout Multi frame Images
github-actions
bot
added
ci/doc
There was a problem hiding this comment.
There's a few linting issues that can be resolved automatically using the make fix[...] targets.
Validate as well using make check[...] targets any other flagged issues that might need some manual adjustments. I didn't comment on all of them.
I did not go in depth into the actual implementation of the transform functions.
I think there is a few tests missing to validate that the actual results from each transformation is valid (eg: pixel values from one format to another are properly rendered or expected text from one format to another is properly structured).
| @@ -298,6 +298,28 @@ install-npm-remarklint: install-npm ## install remark-lint dependency for 'ch | |||
| remark-preset-lint-markdown-style-guide \ | |||
| ) | |||
|
|
|||
| y.PHONY: install-transform | |||
| @@ -28,5 +28,5 @@ services: | |||
| links: | |||
| - mongodb | |||
| volumes: | |||
| - ../../config/weaver.ini.example:/opt/local/src/weaver/config/weaver.ini | |||
| - ../../config/weaver.ini:/opt/local/src/weaver/config/weaver.ini | |||
There was a problem hiding this comment.
Revert this.
We want the example to be tested to ensure it is valid for documentation purpose.
| # noqa: E241,E501 | ||
| (uri_provider_jobs, Visibility.PUBLIC, self.user_admin_id, filter_service(public_jobs)), # noqa: E241,E501 | ||
|
|
||
| ] # type: List[Tuple[str, str, Union[None, int], List[Job]]] |
There was a problem hiding this comment.
Revert auto-formating by your IDE here and above.
| except Exception as e: | ||
| print(cmt + " -> " + wmt + " failed") | ||
| assert False, os.path.splitext(f)[1] + " -> " + f + " " + str(e) | ||
| pass |
|
|
||
|
|
||
|
|
| super(BuiltinProcessJobBase, self).run(runtime_context, **kwargs) | ||
| super(BuiltinProcessJobBase, self).process(runtime_context, **kwargs) |
There was a problem hiding this comment.
Not sure about this change. Looks like an error. It should refer to the CWL's run method I believe.
| Retrieve the possible formats of an output job. | ||
| """ | ||
| job = get_job(request) | ||
| return get_job_possible_output_formats(job) |
There was a problem hiding this comment.
Should do some kind of sd.JobTranformerSchema().deserialize(<body>) check before returning the contents to ensure they match the format of weaver/wps_restapi/examples/job_transformer.json
| def main(): | ||
| image = Image.open('examples/angular.png').convert('RGBA') | ||
| svg_image = rgba_image_to_svg_contiguous(image) | ||
| # svg_image = rgba_image_to_svg_pixels(image) | ||
| with open("examples/angular.svg", "w") as text_file: | ||
| text_file.write(svg_image) | ||
|
|
||
|
|
||
| if __name__ == '__main__': | ||
| main() |
There was a problem hiding this comment.
This looks like something that should be removed.
There was a problem hiding this comment.
Add missing docstrings and typings.
| self.file_path = file_path | ||
| self.cmt = current_media_type.lower() | ||
| self.wmt = wanted_media_type.lower() | ||
| self.output_path = self.file_path |
There was a problem hiding this comment.
The output path needs to be modifiable (either from init arg, or modified after before the process/get calls.
More importantly, no additional files should be added under the original WPS outputs sub-directories, because their contents could be critical to the packing/detection of other apps in a workflow chain.
We need to consider the creation of some "sibling" directory next to the WPS-output-ID dir to avoid modifying the output checksums.
fixes #18
fixes #100
Pour les conversions image, si GeoTiff, distinguo fait dans class Tiff