Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support for parallelisation of analysis (utilise several CPUs) #2803

Open
shakaran opened this issue May 25, 2017 · 8 comments · May be fixed by #7777
Open

Support for parallelisation of analysis (utilise several CPUs) #2803

shakaran opened this issue May 25, 2017 · 8 comments · May be fixed by #7777
Labels
kind/feature request topic/core Core features of Fixer's engine

Comments

@shakaran
Copy link
Contributor

I am noticing in big php projects that run some filters will take hours (mostly when a lot config arrays are in place, for example projects that include other libs inside his source code, like tcpdf in older versions) that takes more than some hours in run the fixers. I notice that it is only using one CPU and I cannot run in parallel, unless that I split several instances in several files or folders at same time.

I would like know if there are a way that the project could accept run in parallel for several cpus (for example one CPU could take the fixer for one file and other CPU for other), so if you have 4 CPUs with 8 threads, you can "fix" 8 files per execution, more quick indeed

@keradus
Copy link
Member

keradus commented May 25, 2017

Hi !
There is no such possibility yet, you are more than welcome to propose solution that would bring that feature !

In the meantime, I suggest to ensure that you are running the newest fixer under PHP 7+ with caching enabled.

@shakaran
Copy link
Contributor Author

shakaran commented May 25, 2017

@keradus thanks! Probably php-cs-fixer could adopt in some way after the param fetching the pcntl_fork(). It duplicates the current PHP process into a child, which can then work separately from the parent process. So per process could init a parsing or rules or fixers.
I am actually using PHP 7.1 with caching

@keradus
Copy link
Member

keradus commented May 25, 2017

I would prefer to use threads instead of processes, using pool pattern and have solution that would provide common interface and hide parallel-implementation thus we rely on same interface regardless parallelization is possible or not.

@GrahamCampbell
Copy link
Contributor

I would prefer to use threads instead of processes, using pool pattern and have solution that would provide common interface and hide parallel-implementation thus we rely on same interface regardless parallelization is possible or not.

👍 For threads. By parallel implementation, what we really mean is, "concurrent" implementation. :trollface: Just pointing out this, for when naming of classes happens.

@keradus
Copy link
Member

keradus commented May 25, 2017

are any of you willing to create POC? if not, I do really doubt that it will happen soon

@shakaran
Copy link
Contributor Author

👍 threads seems more suitable.

I think that the POC could be built around the FixCommand just before to launch the "Runner" class with all the arguments fetched.

Probably the Runner class could be in fact a thread or inherit form a Worker/Thread customized class. So a method like "Divide and Conquer" for "attack" all the files and folders in a directory given (and stop when a file parse is unique as stop condition). So a maximum_threads could be added in configuration per default.

@vertexvaar
Copy link

Since pcntl_fork is not enabled in php by default and real multi-threading is mostly never installed i would suggest using proc_open which can spawn other processes which run concurrently. The nicest and easiest way to achieve this is using symfony/process. As a working example i would bring (JakubOnderka/PHP-Parallel-Lint)[https://github.com/JakubOnderka/PHP-Parallel-Lint] into this. I you are interested in a simple task manager: I have contributed long time ago to a project which acts as a process manager: jagandecapri/symfony-parallel-process.

I hope this helps 😉

@mfn mfn mentioned this issue Dec 23, 2020
AlbertoBarba added a commit to AlbertoBarba/PHP-CS-Fixer that referenced this issue Jul 20, 2021
AlbertoBarba added a commit to AlbertoBarba/PHP-CS-Fixer that referenced this issue Jul 23, 2021
AlbertoBarba added a commit to AlbertoBarba/PHP-CS-Fixer that referenced this issue Jul 27, 2021
AlbertoBarba added a commit to AlbertoBarba/PHP-CS-Fixer that referenced this issue Jul 29, 2021
@Wirone Wirone added kind/feature request status/help wanted Author requests help and removed kind/question status/help wanted Author requests help labels Mar 10, 2023
@Wirone Wirone changed the title [QUESTION][PERFOMANCE] Run php-cs-fixer in parallel (several CPUs) Support for parallelisation of analysis (utilise several CPUs) May 8, 2023
@github-actions

This comment was marked as outdated.

@Wirone Wirone added this to the long-term-ideas milestone Aug 8, 2023
@Wirone Wirone linked a pull request Jan 30, 2024 that will close this issue
8 tasks
@Wirone Wirone added the topic/core Core features of Fixer's engine label Jan 30, 2024
@Wirone Wirone removed this from the long-term-ideas milestone Jan 31, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
kind/feature request topic/core Core features of Fixer's engine
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants