Make "paths that should not be sorted" configurable #704

TravisCarden · 2021-03-05T15:05:44Z

Some elements of composer.json are highly sensitive to order, such as extra.patches with https://github.com/cweagans/composer-patches, which applies patches in the order specified and can therefore fail if it changes. See another example #699. I need a way to exclude arbitrary paths from sorting. I see that the current exclusions are hardcoded in \Ergebnis\Json\Normalizer\Vendor\Composer\ConfigHashNormalizer::PROPERTY_PATHS_THAT_SHOULD_NOT_BE_SORTED. I would like this value to be configurable like the other options. I could take a stab at a PR. Alternatively, a few specific additions to the current hardcoded list would temporarily unblock my customers. I would gladly send a PR for that, too.

extra.installer-paths (Fix: Adjust ComposerJsonNormalizer to stop sorting extra.installer-paths json-normalizer#777)
extra.patches (Fix: Adjust ComposerJsonNormalizer to stop sorting direct children of extra.patches json-normalizer#780)
scripts.auto-scripts (Fix: Adjust ComposerJsonNormalizer to stop sorting scripts.auto-scripts json-normalizer#776)

The text was updated successfully, but these errors were encountered:

localheinz · 2021-03-06T13:57:48Z

@TravisCarden

Which property paths are these?

TravisCarden · 2021-03-08T16:25:16Z

@localheinz Which property paths should be added to the hardcoded exclusions? Off the top of my head I know of the following order-dependent paths:

extra.patches is the one specifically blocking my customers.
extra.installer-paths was mentioned in Composer install paths should not be alphabetized #699.
repositories already seems to be excluded.

rafatwork · 2021-05-17T20:23:08Z

Also using https://github.com/cweagans/composer-patches and experiencing the same issue: patches are re-ordered, but some of them need to be applied in a specific order. I'm currently working around this issue by defining my patches in a separate file (explained @ https://github.com/cweagans/composer-patches#using-an-external-patch-file).

FlorentTorregrosa · 2021-09-01T15:55:03Z

Hello,

About composer-patches, a workaround is to use put number at the beginning of the description like in

            "drupal/core": {
                "1 - BrowserTestBase::setUp() must be compatible with PHPUnit\\Framework\\TestCase::setUp() - https://www.drupal.org/project/drupal/issues/3182103": "https://git.drupalcode.org/project/drupal/-/merge_requests/28.patch"
            }

Best regards,

benr77 · 2022-01-28T16:13:31Z

I also am experiencing problems with this in the "scripts" section - I define various commands to be run via the install and update hooks (via the "auto-scripts" block shown below), but these commands need to be in a specific order, whereas when normalized they are placed in alphabetical order which is not what we want!

"scripts": {
    "post-install-cmd": [
      "@auto-scripts"
    ],
    "post-update-cmd": [
      "@auto-scripts"
    ],
    "auto-scripts": {
      "cache:clear": "symfony-cmd",
      "doctrine:migrations:migrate -v": "symfony-cmd",
      "bazinga:js-translation:dump assets --merge-domains --format=json": "symfony-cmd",
      "assets:install %PUBLIC_DIR%": "symfony-cmd"
    }
  }

When normalized changes the order and becomes:

"scripts": {
    "post-install-cmd": [
      "@auto-scripts"
    ],
    "post-update-cmd": [
      "@auto-scripts"
    ],
    "auto-scripts": {
      "assets:install %PUBLIC_DIR%": "symfony-cmd",
      "bazinga:js-translation:dump assets --merge-domains --format=json": "symfony-cmd",
      "cache:clear": "symfony-cmd",
      "doctrine:migrations:migrate -v": "symfony-cmd"
    }
  }

localheinz · 2022-01-30T16:09:04Z

@benr77 Working on this!

localheinz · 2022-01-31T06:52:21Z

@benr77

I am releasing a temporary fix (see #875) as ergebnis/composer-normalize:2.23.1.

Let's keep this issue open until I have solved the problem!

I apologize for any problems I have caused, and I thank you for opening these issues! Without your help, I would never even know about how different projects use composer.json, and I am learning a lot!

benr77 · 2022-01-31T07:09:15Z

Thanks for your efforts - it really is appreciated a lot!

One point to consider - I see in the PR you are referring to auto-scripts but this is just what I have called the group - I think you can call it anything (this is just a Symfony convention IIRC).

In the scripts block at least, it is OK to sort the first level of children, but anything past that should not be sorted.

Cheers!

localheinz · 2022-01-31T07:13:07Z

Thank you, @benr77!

FlorentTorregrosa · 2022-02-12T18:05:15Z

Hi,

@localheinz don't blame yourself, this tool is amazing! Thanks for providing it in open source with free usage possible!

ergebnis-bot · 2022-08-12T12:05:31Z

Since this issue has not had any activity within the last 180 days, I have marked it as stale.

I will close it if no further activity occurs within the next 14 days.

benr77 · 2022-08-12T14:49:43Z

I believe this issue is still unresolved, at least in the correct manner. Can we not close it please.

Bilge · 2022-11-08T19:53:54Z

I also cannot use this until scripts order normalization can be disabled.

localheinz · 2022-12-12T07:11:17Z

@Bilge

Can you share an example composer.json where you are experiencing issues with the scripts section, please?

orginal
normalized
expected normalized

Thank you!

localheinz · 2022-12-12T07:11:49Z

@benr77

Fixing scripts.auto-scripts with ergebnis/json-normalizer#776.

localheinz · 2022-12-12T07:15:15Z

@TravisCarden

Fixing extra.installer-paths with ergebnis/json-normalizer#777.

Bilge · 2022-12-12T10:04:38Z

@localheinz I just don't want the scripts section "normalized" (alphabetized) at all. I would like to except the scripts section altogether. Same for scripts-descriptions.

localheinz · 2022-12-12T16:33:54Z

@TravisCarden

Regarding extra.patches, can the direct children of extra.patches safely be sorted, or is that also an issue?

Original `composer.json`

{
  "extra": {
    "patches": {
      "drupal/foo": {
        "foo": "https://www.drupal.org/files/issues/2022-01-01/bar.patch",
        "bar": "https://www.drupal.org/files/issues/2020-12-13/foo.patch"
      },
      "drupal/bar": {
        "bar": "https://www.drupal.org/files/issues/2022-01-01/bar.patch",
        "baz": "https://www.drupal.org/files/issues/2019-01-01/baz.patch"
      }
    }
  }
}

Normalized `composer.json`

{
  "extra": {
    "patches": {
      "drupal/bar": {
        "bar": "https://www.drupal.org/files/issues/2022-01-01/bar.patch",
        "baz": "https://www.drupal.org/files/issues/2019-01-01/baz.patch"
      },
      "drupal/foo": {
        "foo": "https://www.drupal.org/files/issues/2022-01-01/bar.patch",
        "bar": "https://www.drupal.org/files/issues/2020-12-13/foo.patch"
      }
    }
  }
}

localheinz · 2022-12-12T16:47:32Z

@FlorentTorregrosa @rafatwork @TravisCarden

Fixing extra.patches with ergebnis/json-normalizer#780.

Does that make sense?

fredden · 2022-12-12T17:16:38Z

Yes, it's safe to sort children of extra.patches, but not their children. That's to say that grandchildren of extra.patches should retain their order.

kAlvaro · 2022-12-12T17:19:57Z

extra is arbitrary data that can be "virtually anything". I don't think a generic tool can make assumptions about its structure. At most, sort direct subkeys.

Regarding scripts, my own use case is like:

    "scripts": {
        "start": [
            "Composer\\Config::disableProcessTimeout",
            "php -S localhost:8080 -t public public/index.php"
        ],
        "phpstan": "phpstan analyse .",
        "phpcs": "phpcs",
        "phpcbf": "phpcbf",
        "rector-lint": "vendor/bin/rector process --dry-run",
        "rector-fix": "vendor/bin/rector process"
    },

Scripts are listed in logical blocks (build actions first, linters last). Linters are shown in order of importance/preference for the project. Each linter displays lint command first, fix command second. Order conveys information, I'm not fond of having it discarded blindly.

On the other side, composer list itself couldn't care less and alphabetises scripts nonetheless.

Would some generic opt-in/opt-out setting be a solution? Can it be flexible enough without adding too much complexity?

{
  "extra": {
    "composer-normalize": {
      "skip-keys": ["scripts", "scripts-descriptions", "extra.patches"]
    }
  }
}

localheinz · 2022-12-12T22:23:38Z

@kAlvaro

From my point of view, JSON objects are objects or maps - if the order of properties or keys matter in an object or a map, then something is broken by design.

I am not judging, but I think it is a bad idea.

ergebnis/json-normalizer only sorts properties of objects, it never sorts lists.

FlorentTorregrosa · 2022-12-13T08:41:13Z

Hi,

@localheinz, about #704 (comment), I confirm, direct children of extra.patches can be sorted (and I like to sort it) but not their own children.

So ok with you comment result.

kAlvaro · 2022-12-13T16:16:26Z

JSON objects are objects or maps - if the order of properties or keys matter in an object or a map, then something is broken by design

You have a point here. Original json.org page says:

An object is an unordered set of name/value pairs".

ECMA-404 says:

The JSON syntax does not impose any restrictions on the strings used as names, does not require that name strings be unique, and does not assign any significance to the ordering of name/value pairs. These are all semantic considerations that may be defined by JSON processors or in specifications defining specific uses of JSON for data interchange

And I don't think Composer has anything to say about order.

I'd still appreciate a generic setting to customise behaviour for custom keys, but it's out of scope.

TravisCarden · 2022-12-13T16:23:47Z

@localheinz++ 😄

localheinz self-assigned this Mar 6, 2021

localheinz added the enhancement label Jan 30, 2022

localheinz mentioned this issue Jan 31, 2022

Fix: Prevent updates of ergebnis/json-normalizer for now #875

Merged

3 tasks

ergebnis-bot added the stale label Aug 12, 2022

ergebnis-bot removed the stale label Aug 13, 2022

This was referenced Nov 8, 2022

Fix test cases for new normalisation rules #985

Closed

Add option to allow certain keys to preserve their order / not be normalised #996

Closed

localheinz mentioned this issue Dec 12, 2022

Fix: Adjust ComposerJsonNormalizer to stop sorting scripts.auto-scripts ergebnis/json-normalizer#776

Merged

1 task

localheinz mentioned this issue Dec 12, 2022

Fix: Adjust ComposerJsonNormalizer to stop sorting extra.installer-paths ergebnis/json-normalizer#777

Merged

1 task

This was referenced Dec 12, 2022

Fix: Adjust ComposerJsonNormalizer to stop sorting config.allow-plugins ergebnis/json-normalizer#778

Merged

Fix: Adjust ComposerJsonNormalizer to stop sorting config.preferred-install ergebnis/json-normalizer#779

Merged

localheinz mentioned this issue Dec 12, 2022

Fix: Adjust ComposerJsonNormalizer to stop sorting direct children of extra.patches ergebnis/json-normalizer#780

Merged

1 task

localheinz linked a pull request Feb 26, 2023 that will close this issue

composer(deps): Bump ergebnis/json-normalizer from 2.1.0 to 4.0.0 #1056

Merged

localheinz closed this as completed in #1056 Feb 26, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make "paths that should not be sorted" configurable #704

Make "paths that should not be sorted" configurable #704

TravisCarden commented Mar 5, 2021 •

edited by localheinz

localheinz commented Mar 6, 2021

TravisCarden commented Mar 8, 2021

rafatwork commented May 17, 2021

FlorentTorregrosa commented Sep 1, 2021

benr77 commented Jan 28, 2022

localheinz commented Jan 30, 2022

localheinz commented Jan 31, 2022 •

edited

benr77 commented Jan 31, 2022

localheinz commented Jan 31, 2022

FlorentTorregrosa commented Feb 12, 2022

ergebnis-bot commented Aug 12, 2022

benr77 commented Aug 12, 2022

Bilge commented Nov 8, 2022

localheinz commented Dec 12, 2022

localheinz commented Dec 12, 2022

localheinz commented Dec 12, 2022

Bilge commented Dec 12, 2022 •

edited

localheinz commented Dec 12, 2022 •

edited

localheinz commented Dec 12, 2022

fredden commented Dec 12, 2022

kAlvaro commented Dec 12, 2022 •

edited

localheinz commented Dec 12, 2022 •

edited

FlorentTorregrosa commented Dec 13, 2022

kAlvaro commented Dec 13, 2022 •

edited

TravisCarden commented Dec 13, 2022

Make "paths that should not be sorted" configurable #704

Make "paths that should not be sorted" configurable #704

Comments

TravisCarden commented Mar 5, 2021 • edited by localheinz

localheinz commented Mar 6, 2021

TravisCarden commented Mar 8, 2021

rafatwork commented May 17, 2021

FlorentTorregrosa commented Sep 1, 2021

benr77 commented Jan 28, 2022

localheinz commented Jan 30, 2022

localheinz commented Jan 31, 2022 • edited

benr77 commented Jan 31, 2022

localheinz commented Jan 31, 2022

FlorentTorregrosa commented Feb 12, 2022

ergebnis-bot commented Aug 12, 2022

benr77 commented Aug 12, 2022

Bilge commented Nov 8, 2022

localheinz commented Dec 12, 2022

localheinz commented Dec 12, 2022

localheinz commented Dec 12, 2022

Bilge commented Dec 12, 2022 • edited

localheinz commented Dec 12, 2022 • edited

Original composer.json

Normalized composer.json

localheinz commented Dec 12, 2022

fredden commented Dec 12, 2022

kAlvaro commented Dec 12, 2022 • edited

localheinz commented Dec 12, 2022 • edited

FlorentTorregrosa commented Dec 13, 2022

kAlvaro commented Dec 13, 2022 • edited

TravisCarden commented Dec 13, 2022

TravisCarden commented Mar 5, 2021 •

edited by localheinz

localheinz commented Jan 31, 2022 •

edited

Bilge commented Dec 12, 2022 •

edited

localheinz commented Dec 12, 2022 •

edited

Original `composer.json`

Normalized `composer.json`

kAlvaro commented Dec 12, 2022 •

edited

localheinz commented Dec 12, 2022 •

edited

kAlvaro commented Dec 13, 2022 •

edited