Add yaml output #41

cbm755 · 2023-02-25T07:54:31Z

Fixes #37

todo

add test
add another test for the req file
~~build_module_as_str seems unused and untested, same with the new build_module_as_yaml_str.~~ Filed classmethod build_module_as_str is untested? #45 for this.
local imports for pyyaml dep, not at top
use file-name to guess output (default json)
add --yaml flag
--platform-info should respect --yaml flag

help wanted

poetry extra dep (I'm not a poetry user, not sure about this
left a TODO

Start of a fix for Issue johannesjh#37.

req2flatpak.py

pyproject.toml

req2flatpak.py

Co-authored-by: yfprojects <62463991+real-yfprojects@users.noreply.github.com>

req2flatpak.py

real-yfprojects · 2023-02-25T18:59:32Z

Are you using pre-commit?

cbm755 · 2023-02-25T23:54:33Z

Are you using pre-commit

~~No I have never learned that, was always worried it would mess up by other projects. I do manually run black before committing, but I understand maybe there are other standards at play here...~~

update luddite-mode turned off, trying it...

I think this is for older pyyaml, AFAICT, False is the default circa 2022. No need to specific indent=2 as that seems to be the default.

cbm755 · 2023-02-26T00:19:46Z

pre-commit mucked around in poetry.lock with lots of unrelated changes. I'll not commit those.

pylama gaves lots of strange-looking errors so I uninstalled that.

poetry-check.............................................................Passed
poetry-lock..............................................................Passed
Sync with Poetry.........................................................Passed
isort (python)...........................................................Passed
black....................................................................Passed
prettier.................................................................Passed
rstcheck.................................................................Passed
pylama...................................................................Failed
- hook id: pylama
- exit code: 1

Executable `pylama` not found

bandit...................................................................Failed
- hook id: bandit
- exit code: 1

[node_visitor]	WARNING	Unable to find qualified name for module: req2flatpak.py
--------------------------------------------------
 tests/test_req2flatpak.py:106:
 [  B506  ] Severity: MEDIUM Confidence: HIGH
 Use of unsafe yaml load. Allows instantiation of arbitrary objects. Consider yaml.safe_load().

The yaml.safe_load() seems nothing to worry about b/c its in a test.

cbm755 · 2023-02-26T00:20:57Z

now I can't commit, not too impressed so far...

[WARNING] Stashed changes conflicted with hook auto-fixes... Rolling back fixes...
[INFO] Restored changes from /home/cbm/.cache/pre-commit/patch1677370833-285058.

cbm755 · 2023-02-26T00:22:11Z

... and installing it does not go well:

git commit
/usr/bin/python3: No module named pre_commit

So back to luddite mode for me I think!

cbm755 · 2023-02-26T00:25:20Z

I'm not very confident about the test: please take a look at that.

real-yfprojects · 2023-02-26T14:39:14Z

So back to luddite mode for me I think!

Don't blame me, but I had to smile reading about your struggle. Our pre-commit setup is not prefect yet. I tweaked it in #42 and #39.

No I have never learned that, was always worried it would mess up by other projects.

You can instal it on a pre-repository basis.

tests/test_req2flatpak.py

real-yfprojects · 2023-02-26T14:45:22Z

To fix the ci you will have to run the following:

poetry lock --no-update && SKIP=poetry-lock git commit

Also avoids warnings from linters.

cbm755 · 2023-02-27T19:42:02Z

This is ready to go from my point-of-view, but maybe needs some clean-up about testing with optional dependency pyyaml (which is not optional for testing).

real-yfprojects · 2023-02-27T19:55:04Z

I can add a commit for fixing ci.

tests/test_req2flatpak.py

* pyproject.toml (tool.poetry.dependencies): Add `pyyaml` as optional dependency. * poetry.lock : Run `poetry lock --no-update` to add pyyaml.

req2flatpak.py

* pyproject.toml * poetry.lock

* .github/workflows/ci.yml

tests/test_req2flatpak.py

johannesjh · 2023-03-05T08:10:42Z

hi guys, this PR is almost finished, well done! we have a few open comments from the code review and three complaints from pylint. @cbm755 are you going to walk the last mile? I hope you are not getting annoyed, do you need / want help?

real-yfprojects · 2023-03-05T08:54:26Z

three complaints from pylint

I don't know about the complexity complaint. The import complaints could be resolved similar to how I did it in the tests.

johannesjh · 2023-03-05T09:30:28Z

I would suggest to ignore the complexity constraint.

The complaint is pylint R0912 "too many branches".
A proper solution would be to refactor the code and split the main methods into multiple methods. I doubt this will make the code easier to understand or maintain.
Let's ignore the complaint and work around the issue by commenting the line with # pylint: disable=too-many-branches.

Yaml is tried to be imported else `yaml` is set to `None`. Later the code will only check wether `yaml` is True (is not None). * req2flatpak.py

* req2flatpak.py : Ignore pylint complain `too-many-branches` for `main`.

* tests/test_req2flatpak.py (Req2FlatpakBaseTest.requirements_file): Implement this method returning a contextmanager that creates and prefills a requirements file with requirements. * tests/test_req2flatpak.py (Req2FlatpakBaseTest.test_cli_with_reqs_as_file[_yaml]): Use `requirements_file()`.

These methods implement formatting, so why duplicate that code. Fixes johannesjh#45. * req2flatpak.py (main)

cbm755 · 2023-03-17T02:50:19Z

Thanks @real-yfprojects for the edits. I tested this and it works for me!

I do notice that the yaml dict out is in a rather unpleasant order:

build-commands:
- pip3 install --verbose --exists-action=i --no-index --find-links="file://${PWD}"
  --prefix=${FLATPAK_DEST} --no-build-isolation pymupdf
buildsystem: simple
name: python3-package-installation
sources:
- only-arches:
  - aarch64
  sha256: 36181c4cb27740791d611d0224dd18ac78e23a1f06aa2151394c5b9194b4f885
  type: file
  url: https://files.pythonhosted.org/packages/9c/90/4dcdb731e9aaecfa2f10300bf51f4f2d2079083819fd4a109a3e1d331338/PyMuPDF-1.21.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl
- only-arches:
  - x86_64
  sha256: e5273ff0c3bf08428ef956c4a5e5e0b475667cc3e1cb7b41d9f5131636996e59
  type: file
  url: https://files.pythonhosted.org/packages/e0/27/83c843ef54f5c65874c844a77e216326eb5b2a2fcb08bd7cb0784fb759f9/PyMuPDF-1.21.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl

Note "build-commands" before "name" for example. And "sha256" before "type". Its still valid but I wonder why that happens? Should I file an new issue?

For comparison, here is the JSON:

{
    "name": "python3-package-installation",
    "buildsystem": "simple",
    "build-commands": [
        "pip3 install --verbose --exists-action=i --no-index --find-links=\"file://${PWD}\" --prefix=${FLATPAK_DEST} --no-build-isolation pymupdf"
    ],
    "sources": [
        {
            "type": "file",
            "url": "https://files.pythonhosted.org/packages/9c/90/4dcdb731e9aaecfa2f10300bf51f4f2d2079083819fd4a109a3e1d331338/PyMuPDF-1.21.1-cp310-cp310-manylinux_2_17_aarch64.manylinux2014_aarch64.whl",
            "sha256": "36181c4cb27740791d611d0224dd18ac78e23a1f06aa2151394c5b9194b4f885",
            "only-arches": [
                "aarch64"
            ]
        },
        {
            "type": "file",
            "url": "https://files.pythonhosted.org/packages/e0/27/83c843ef54f5c65874c844a77e216326eb5b2a2fcb08bd7cb0784fb759f9/PyMuPDF-1.21.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl",
            "sha256": "e5273ff0c3bf08428ef956c4a5e5e0b475667cc3e1cb7b41d9f5131636996e59",
            "only-arches": [
                "x86_64"
            ]
        }
    ]
}

My understanding of modern Python is that dicts have a consistent unofficial ordering based on the the order you insert things into them. In this case, that translates into a pleasant ordering when outputing JSON but not YAML...

cbm755 · 2023-03-17T03:48:37Z

I found a very long thread about it here where I discovered there is a sort_keys=False option.

real-yfprojects · 2023-03-17T15:09:35Z

I do notice that the yaml dict out is in a rather unpleasant order:

Good call!

johannesjh · 2023-03-18T15:32:47Z

well done, thank you for the effort! ready to merge. 🎉

cbm755 added 2 commits February 24, 2023 23:48

Consistent json indent

8518ee3

Write yaml if output file ends in yaml or yml extension

2ea3d70

Start of a fix for Issue johannesjh#37.

real-yfprojects reviewed Feb 25, 2023

View reviewed changes

req2flatpak.py Show resolved Hide resolved

pyproject.toml Outdated Show resolved Hide resolved

req2flatpak.py Outdated Show resolved Hide resolved

cbm755 and others added 3 commits February 25, 2023 08:51

Update pyproject.toml

1b04329

Co-authored-by: yfprojects <62463991+real-yfprojects@users.noreply.github.com>

Add a "--yaml" cmdline arg

3a2f60d

write yaml from '--platform-info' too

db2918c

real-yfprojects reviewed Feb 25, 2023

View reviewed changes

req2flatpak.py Outdated Show resolved Hide resolved

req2flatpak.py Outdated Show resolved Hide resolved

Refactor yaml

2aa87c4

cbm755 force-pushed the yaml branch from 803e802 to 2aa87c4 Compare February 25, 2023 18:49

Force default flow style to False

6f622a8

I think this is for older pyyaml, AFAICT, False is the default circa 2022. No need to specific indent=2 as that seems to be the default.

Add test for yaml output

820c43b

real-yfprojects reviewed Feb 26, 2023

View reviewed changes

tests/test_req2flatpak.py Outdated Show resolved Hide resolved

cbm755 mentioned this pull request Feb 27, 2023

classmethod build_module_as_str is untested? #45

Closed

cbm755 added 2 commits February 26, 2023 19:59

Add another yaml test for requirements file

df7bea2

test: use yaml safe_load which is simpler

126f9b8

Also avoids warnings from linters.

real-yfprojects reviewed Feb 28, 2023

View reviewed changes

tests/test_req2flatpak.py Outdated Show resolved Hide resolved

Fix extra dependency configuration and add pyyaml to lock file.

377a4a4

* pyproject.toml (tool.poetry.dependencies): Add `pyyaml` as optional dependency. * poetry.lock : Run `poetry lock --no-update` to add pyyaml.

johannesjh mentioned this pull request Feb 28, 2023

Fix invalid platforms in main. #39

Merged

johannesjh requested changes Feb 28, 2023

View reviewed changes

req2flatpak.py Outdated Show resolved Hide resolved

req2flatpak.py Outdated Show resolved Hide resolved

Add types-pyyaml to lint dev packages.

f5ae4fe

* pyproject.toml * poetry.lock

real-yfprojects force-pushed the yaml branch 9 times, most recently from 92f1ea4 to 72109fd Compare February 28, 2023 13:18

Generate test matrix dynamically for extras.

de6c5a6

* .github/workflows/ci.yml

real-yfprojects force-pushed the yaml branch from 72109fd to de6c5a6 Compare February 28, 2023 13:19

real-yfprojects reviewed Feb 28, 2023

View reviewed changes

tests/test_req2flatpak.py Show resolved Hide resolved

real-yfprojects added 4 commits March 11, 2023 19:53

Move import of yaml to the beginning of req2flatpak.py.

0f42bc4

Yaml is tried to be imported else `yaml` is set to `None`. Later the code will only check wether `yaml` is True (is not None). * req2flatpak.py

Ignore too-many-branches in main function.

d27d35e

* req2flatpak.py : Ignore pylint complain `too-many-branches` for `main`.

Use FlatpakGenerator.build_module_as_[yaml_]str in main.

6d9303d

These methods implement formatting, so why duplicate that code. Fixes johannesjh#45. * req2flatpak.py (main)

real-yfprojects force-pushed the yaml branch from de6e865 to 6d9303d Compare March 11, 2023 19:20

real-yfprojects requested a review from johannesjh March 11, 2023 19:24

Fix dict-ordering when outputing yaml

ed86921

johannesjh approved these changes Mar 18, 2023

View reviewed changes

johannesjh merged commit bfa9f75 into johannesjh:main Mar 18, 2023
7 checks passed

cbm755 deleted the yaml branch March 21, 2023 05:14

johannesjh added this to the v0.2 milestone Apr 5, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add yaml output #41

Add yaml output #41

cbm755 commented Feb 25, 2023 •

edited

real-yfprojects commented Feb 25, 2023

cbm755 commented Feb 25, 2023 •

edited

cbm755 commented Feb 26, 2023

cbm755 commented Feb 26, 2023

cbm755 commented Feb 26, 2023

cbm755 commented Feb 26, 2023

real-yfprojects commented Feb 26, 2023

real-yfprojects commented Feb 26, 2023

cbm755 commented Feb 27, 2023

real-yfprojects commented Feb 27, 2023

johannesjh commented Mar 5, 2023

real-yfprojects commented Mar 5, 2023

johannesjh commented Mar 5, 2023

cbm755 commented Mar 17, 2023

cbm755 commented Mar 17, 2023

real-yfprojects commented Mar 17, 2023

johannesjh commented Mar 18, 2023

Add yaml output #41

Add yaml output #41

Conversation

cbm755 commented Feb 25, 2023 • edited

todo

help wanted

real-yfprojects commented Feb 25, 2023

cbm755 commented Feb 25, 2023 • edited

cbm755 commented Feb 26, 2023

cbm755 commented Feb 26, 2023

cbm755 commented Feb 26, 2023

cbm755 commented Feb 26, 2023

real-yfprojects commented Feb 26, 2023

real-yfprojects commented Feb 26, 2023

cbm755 commented Feb 27, 2023

real-yfprojects commented Feb 27, 2023

johannesjh commented Mar 5, 2023

real-yfprojects commented Mar 5, 2023

johannesjh commented Mar 5, 2023

cbm755 commented Mar 17, 2023

cbm755 commented Mar 17, 2023

real-yfprojects commented Mar 17, 2023

johannesjh commented Mar 18, 2023

cbm755 commented Feb 25, 2023 •

edited

cbm755 commented Feb 25, 2023 •

edited