Composer can't find unzip or 7zip when run via php+apache #11923
Comments
|
I could be thinking/looking at everything out of context so please let me know if any of the below is wrong. Looking at composer source I think like it is using symfony/process v5.4.36. This worried me at first knowing that there is a v7 out there, but looking at the releases it appears they keep a stable branch of 5, 6, and 7. Looking at the v5.4 branch I copied the code and ran it in my test script. function find(string $name, ?string $default = null, array $extraDirs = [])
{
if (\ini_get('open_basedir')) {
$searchPath = array_merge(explode(\PATH_SEPARATOR, \ini_get('open_basedir')), $extraDirs);
$dirs = [];
foreach ($searchPath as $path) {
// Silencing against https://bugs.php.net/69240
if (@is_dir($path)) {
$dirs[] = $path;
} else {
if (basename($path) == $name && @is_executable($path)) {
return $path;
}
}
}
} else {
$dirs = array_merge(
explode(\PATH_SEPARATOR, getenv('PATH') ?: getenv('Path')),
$extraDirs
);
}
$suffixes = [''];
if ('\\' === \DIRECTORY_SEPARATOR) {
$pathExt = getenv('PATHEXT');
$suffixes = array_merge($pathExt ? explode(\PATH_SEPARATOR, $pathExt) : $this->suffixes, $suffixes);
}
foreach ($suffixes as $suffix) {
foreach ($dirs as $dir) {
if (@is_file($file = $dir.\DIRECTORY_SEPARATOR.$name.$suffix) && ('\\' === \DIRECTORY_SEPARATOR || @is_executable($file))) {
return $file;
}
}
}
return $default;
}
var_dump(find('unzip'));and this output
which makes me think the checking of the executable is working fine. Going back to the composer source I can't see why this would not run correclty. Even $hasSystemUnzip = (bool) find('unzip');
var_dump($hasSystemUnzip);outputs
Running from source vs running from what I dowonload from here might be different, but I don't see what that could be. Any help would be appreciated. |
|
I was able to re-produce this on a fresh Amazon Linux 2 as well as Amazon Linux 2023 Setup code for Amazon Linux 2 And setup instructions for a fresh Amazon Linux 2023, Going to both in browser with Amazon Linux 2 Amazon Linux 2023 Same EDIT: Other env where I thought things were without warnings is in fact not without warnings and it gives me the follwing when installing composer packages. (older composer, older php) Diagnostics on this server, I am going to have to see if I have logs to dig into to see if this has always been the case. I don't think it has beem but I could be wrong. In either case it shouldn't be happening as the required packages are installed. EDIT 2: Running that on our Amazon Linux 1 server we are updating from
👀 EDIT 3: Amazon Linux 2, default out of the box php (5.4.16). I can't install php 7 from Amazon Linux Extras, From this I can only see that things don't work as expected on php8, but seem fine on php5 and php7. |
|
The only thing I can think of is that the PATH env var somehow gets messed up/restricted in some env/cases. What exactly causes that I have no clue though, I am not familiar at all with Amazon Linux sorry. Note that for git we never actually check its presence using the executable finder, we just attempt to execute it. |
|
Is there any commands that make composer output it's known env by any chance? If not I can look at what a phar is and making my own to do just that. I'll also double check I can't replicate on Ubuntu just to be sure. |
|
You can just clone this repo and run bin/composer instead of the phar to debug stuff. Or run bin/compile to make your own phar if that's easier to get deployed |
|
Aighty, so I got that running. I setup a basic foreach (getenv() as $key => $value)
{
echo "$key - $value<br />";
}which outputs, From a seperate script I call with I modified the bottom of composer iiagnose to have the following foreach (getenv() as $key => $value)
{
$io->write("$key - $value");
}
$this->runCommand($io, 'unzip -v');
....
private function runCommand($io, $command)
{
$tmp = array();
exec($command.' 2>&1', $tmp, $return_code);
$io->write('command: '.$command);
$io->write('return_code: '.$return_code);
$io->write('output:');
foreach ($tmp as $output_line)
{
$io->write($output_line);
}
$io->write('');
}which I executed with runCommand('/var/www/vhost/mySite.com/composer-2.7.2/bin/composer diagnose -vvv');and the bottom part of that output was, My takeaway from that is composer can use unzip, but the symfony code to detect unzip does not work. I'll continue down this rabbit hole and make a standalone bit of code for symfony and play around with the detection. I'll also make an Ubuntu webserver and replicate it as I mentioned above. |
|
Ok this is very odd, if there is no PATH available I don't know how just executing |
|
Ah, I think that may have been my bad. The runCommand('echo $PATH');So the PATH does contain the things it would expect. I still need to compare against Ubuntu and see if it's an issue with composer running via php-fpm via apache. I think I will have to go take all of this to symfony, as it now looks like it's their code that is failing to detect executables and composer is just (understandably) going along with it. |
|
So if you do |
|
Both var_dump($_SERVER['PATH']);and var_dump(getenv('PATH'));output But foreach (getenv(null) as $key => $value)
{
echo "$key - $value<br />";
}only outputs
which is odd because unless I am misunderstanding the docs it should output them all. EDIIT: Woah.. this is odd var_dump($_SERVER['PATH']);
echo '<br />';
foreach (getenv(null) as $key => $value)
{
echo "$key - $value<br />";
}
echo "<br />";I would expect just the stuff we saw above, but nope! It's almost like Testing composer diag after the Posted that to Stack Overflow though because I am confused why that is. |
|
This is most likely because of the
I guess getenv does not trigger initialization of $_SERVER. That sounds like a bug to me.. but anyway that's another problem. I also cannot reproduce this within a CLI context. I am not sure how you run composer exactly but it looks to be through an apache web request, so some things may be different in that context 🤷🏻 |
|
Confirmed, setting Back to the problem. It's odd that I also can't replicate via CLI, only when script is run via apache. I am now curious if running composer via a webhook is just not very common, and so long as you have php-zip installed it will still work fine so people havn't realised it or brought it up. OR its something very specific to my situation 😅 I havn't had time yet but when I do I'll do the following and get back to this thread with more info.
Now that I have composer running from source I can also try these
|
|
Using composer within a web SAPI is definitely not the most common, although some others do it. I'm not sure if they've run into issues with unzip not being detected though. Maybe @Toflar has a clue here. |
|
And btw the ZipDownloader does also check which unzip software is available using ExecutableFinder, so if diagnose reports it is missing it should not be using it at install time. |
|
Forgive me if I did not get the point of this issue - my time is too limited to study all of your comments but just from skimming this issue I can tell that you somehow compare the web request to the CLI request? |
|
I see, then you probably would not have run in the issue here (TLDR: unzip is not detected when running in apache SAPI, because somehow the PATH is not set correctly? Or something 🤷🏻) |
|
I suspect @beeradmoore is using |
|
From my understanding we are not using I am not sure about pathing and other env, but it isn't 100% the apache user as running CLI as my user, or as apache gives the same result of zip being detected.
Looking at our older php5/7 servers I think they were using I just did one of the tests I mentioned above and modified public function find(string $name, ?string $default = null, array $extraDirs = [])
{
if ($name == 'unzip')
{
return '/usr/bin/unzip';
}
...and then run my normal deploy script and it worked perfectly fine. With that info I would be happy to stop here and say that this issue is 100% a symfony problem (EDIT: reproduction of symfony outside of composer and it is able to find unzip :| ). I wouldn't expect composer to re-write its process finding code to do its own thing instead of relying on symfony. I'll go reproduce in a smaller example and file an issue there and see what comes of it. If they were able to fix it on their end what is the process to get that in composer? Is there a set release schedule, is there anything stopping composer from doing a version bump, etc Thanks heaps for the responses above, helped me figure out how to debug it further. |
|
Can you check the following differences between your web and CLI setups (make sure you log in as the respective user):
|
|
Reproducing the issue didn't go well, test.php <?php
require_once '/var/www/vhost/composer-2.7.2/vendor/autoload.php';
use Symfony\Component\Process\ExecutableFinder;
$finder = new ExecutableFinder;
$hasSystemUnzip = (bool) $finder->find('unzip');
var_dump($hasSystemUnzip);But if I try something like this in a web request exec('php test.php', $tmp, $return_code);also reports in $tmp that it got Considering that same code within composer executed in (as far as I am aware, the exact same way) I would have expected that second one to incorrectly report false. As for the things you asked for
Via request in browser <?php
var_dump(ini_get('open_basedir'));Also request in browser, <?php
var_dump($_SERVER['PATH']); |
|
Composer restarts itself in case xdebug is activated - probably this might be another hint? See https://github.com/composer/xdebug-handler |
|
Oh, thanks for that link. That is very handy. Things are starting to fall together. When composer restarts I think it no longer has output of that which is important The bit that falls into place is from from the tests above and I was curious that
So I am about to stop for the night, but when I debug this some more soon I'll see if I can force composer to not restart and if that works I'll see if I can figure out why |
|
Yeah if this gets fixed in symfony 5.4.x (which it should if it can be fixed, but I'm still not sure it isn't an environment problem) we will get the fix in the next patch release. |
|
You can force Composer not to restart by setting Also, it's not all Symfony Process. Composer's XdebugHandler directly uses |
|
Sorry, I missed the second half of this,
I don't have xdebug enabled. This server may have had it enabled at some point. In previous testing above I started a new server and installed everything from scratch so I know it didn't have xdebug there. I'll dig into that and see if it's restarting when it shouldn't be. |
|
Set |
|
Hey @johnstevenson , any idea where I'd expect for php to send stderr to? I assumed <?php
putenv('COMPOSER_ALLOW_XDEBUG=1');
putenv('XDEBUG_HANDLER_DEBUG=1');
function runCommand($command)
{
$tmp = array();
exec($command.' 2>&1', $tmp, $return_code);
echo 'command: '.$command.'<br>';
echo 'return_code: '.$return_code.'<br>';
echo 'output: <br>';
foreach ($tmp as $output_line)
{
echo $output_line.'<br>';
}
echo '<br>';
}
runCommand('/var/www/vhost/composer-2.7.2/bin/composer diagnose -vvv');
I also tried runCommand('COMPOSER_ALLOW_XDEBUG=1 XDEBUG_HANDLER_DEBUG=1 /var/www/vhost/composer-2.7.2/bin/composer diagnose -vvv'); |
|
There will only be output if |
EDIT: From the bottom of all these tests and various comments it seems that version of Amazon Linux (1, 2, 2023) does not matter, and this is a php5/7 vs php8 issue.
We use a custom php script to run
composer install. While migrating our systems to Amazon Linux 2023 we started getting some warnings in these composer installs which have worked on both Amazon Linux 1 and Amazon Linux 2 instances for some time now.While trying to run this update script we get this output
Note
I understand this may be an issue with something new that has changed in Amazon Linux 2023 or even a problem with Symfony and how it checks for installed package rather than a problem with composer itself, but I am just trying to cover all bases.
The problem with the above warnings is that both unzip and 7zip (7zz) are installed.
In a test script I have made I have a php function that executes commands, I can run this in a browser to reproduce the issue we are having. That function looks like this,
Running
unzip -vgivesRunning
7zzgives,But the output of
composer diagnose -vvvis,This here is also reproting that unzip and 7-Zip are not avaialble.
But even above it was able to see and execute git and see that it is available.
I also wondered if things are installed in other places and there is an access issue.
So I am at a loss as to why I can execute
unzipand7zzbut executing composer seems to think they are not avaialbe.Additionally running diagnose from command line as the apache user gives full green lights.
EDIT: Disabling SELinux did not fix the issue.
The text was updated successfully, but these errors were encountered: