Optimization opportunities for composer metadata loading

[stof]

@javiereguiluz made some analysis of a run of composer in symfony/skeleton#66 (comment) (he can probably share the corresponding blackfire profile).

Regarding the strtolower calls, an idea might be to make a special case for Packagist repositories (with an opt-in for any other custom repo of course, but it would be enabled for the builtin repo), which could rely on the fact that package names are already lowercased in the downloaded metadata (I'm quite sure that's already the case, and it would not be hard to implement otherwise).
This way, composer would not need to process the package names to make them lowercase.

[javiereguiluz]

To give more context about this. We were discussing about reducing installation time for a PHP app that has these simple composer.json requirements:

    "require": {
        "php": "^7.1.3",
        "ext-ctype": "*",
        "ext-iconv": "*",
        "symfony/console": "^4.0",
        "symfony/flex": "^1.0",
        "symfony/framework-bundle": "^4.0",
        "symfony/lts": "^4@dev",
        "symfony/yaml": "^4.0"
    },
    "require-dev": {
        "symfony/dotenv": "^4.0"
    },

I mentioned that it's sad that JavaScript's yarn is so fast and Composer is a bit slow. But I also said that it's not Composer's maintainers fault! I love the work you do. This should be a global effort: companies using PHP should donate money to this initiative and people from all around the world should step up and help you!

In any case, I profiled a simple composer create symfony/skeleton install and it was surprising to me some things, like calling to strotolower() almost half a million times! This is the full profile: https://blackfire.io/profiles/c65ba07d-1e0d-4c74-ad25-da858fa8cd8e/graph

[alcohol]

Please, try to keep comparisons / mentions of other package managers of other languages out of the discussion.

[javiereguiluz]

Thanks for the advice ... but I recommend to do the opposite. Please, look at many other package managers as possible to get great ideas from them. There will be obvious unavoidable differences because each programming language has its own features and some of them can't be translated to PHP. But other than that, please look for the great ideas implemented by other projects. Thanks!

[alcohol]

I meant mentions like these:

I mentioned that it's sad that JavaScript's yarn is so fast and Composer is a bit slow.

This contributes nothing useful to the discussion.

[javiereguiluz]

I think it was useful because it can lead to massive improvements. For example, when running composer create symfony/skeleton, Composer made 72 non-parallel HTTP requests (see symfony/skeleton#66 (comment)). Then I run JavaScript's Yarn and I see that it's blazing-fast. Why? Among other things, because it makes parallel HTTP requests. That's the kind of ideas we could take from other package managers. Thanks!

[alcohol]

See, that is actually useful. Talking about features / implementation details. Parallel requests are already possible through some extensions/plugins. I am pretty sure they are also already on the roadmap for Composer 2.x as well.

I was just saying that comments like "X is fast and Composer is slow" do not help.

[archon810]

Parallel downloads are needed so badly. Our project that uses a lot of Wordpress plugins could take 5 minutes for composer update to complete, all because it downloads many repos from wpackagist one by one. Is there a better ticket to watch for parallel downloads implementation?

[alcohol]

Not really. You can already use https://github.com/hirak/prestissimo in the meanwhile.

[archon810]

Not really. You can already use https://github.com/hirak/prestissimo in the meanwhile.

I love you!

[stof]

Please keep this issue focused. Parallel downloads are totally unrelated to the optimization suggested here (there are other issues discussing them)

[Seldaek]

Composer 2 is coming 🥳