Different results with similar setup in different environments

[fkupper]

I am running psalm on my project as a composer dev dependency on my machine in a VM provided by Homestead and I also run it on circleci on every push.

Since I have introduced it to this project not long ago and there is a lot of legacy code, I have decided to use a baseline file.

All was working fine until I stopped using it as a composer dev (version 3.0.2) dependency and moved it to a global composer install (version dev-master). The reason for this was because I was having conflicts with other libraries I use and the latest version of PSALM was not compatible, and that is fixable by running it globally since I have no other global dependency.

I have updated my baseline file for the latest version of PSALM as it was complaining about new errors and all was good, but running it on circleci yields different results from my dev environment.

This is my config file:

<?xml version="1.0"?>
<psalm
    useDocblockTypes="true"
    totallyTyped="false"
    errorBaseline="psalm-baseline.xml"
>
  <projectFiles>
      <ignoreFiles>
          <directory name="vendor"/>
      </ignoreFiles>
      <file name="bootstrap/app.php" />
      <directory name="app"/>
      <directory name="routes"/>
      <directory name="libraries"/>
  </projectFiles>
  <issueHandlers>
        <MissingClosureReturnType errorLevel="info"/>
        <MissingPropertyType errorLevel="info"/>
        <PossiblyUnusedMethod errorLevel="info"/>
        <InvalidDocblock errorLevel="info"/>
        <PropertyNotSetInConstructor errorLevel="info"/>
        <MissingConstructor errorLevel="info"/>
  </issueHandlers>
</psalm>

I am running PSALM with the following options:

~/.composer/vendor/bin/psalm --config="psalm.xml" --show-info=false --use-ini-defaults --find-dead-code

CircleCI output

Scanning files...
Analyzing files...

░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░  60 / 349 (17%)
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 120 / 349 (34%)
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 180 / 349 (51%)
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 240 / 349 (68%)
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 300 / 349 (85%)
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░

<I am omitting the reported errors>

------------------------------
15 errors found
------------------------------
3852 other issues found.
------------------------------
Psalm can automatically fix 614 of them.
Run Psalm again with
--alter --issues=MissingReturnType,UnusedVariable,InvalidReturnType,InvalidNullableReturnType,MismatchingDocblockReturnType,MismatchingDocblockParamType,InvalidFalsableReturnType,MissingParamType,UnusedMethod --dry-run
to see what it can fix.
------------------------------

Checks took 15.08 seconds and used 962.596MB of memory
Psalm was able to infer types for 69.8374% of the codebase

Dev environment output

Scanning files...
Analyzing files...

░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░  60 / 349 (17%)
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 120 / 349 (34%)
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 180 / 349 (51%)
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 240 / 349 (68%)
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░ 300 / 349 (85%)
░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░░

------------------------------
No errors found!
------------------------------
3855 other issues found.
------------------------------
Psalm can automatically fix 614 of them.
Run Psalm again with 
--alter --issues=MissingReturnType,InvalidNullableReturnType,UnusedVariable,InvalidReturnType,MismatchingDocblockReturnType,InvalidFalsableReturnType,MismatchingDocblockParamType,UnusedMethod,MissingParamType --dry-run
to see what it can fix.
------------------------------

Checks took 61.13 seconds and used 646.082MB of memory
Psalm was able to infer types for 69.7228% of the codebase

What I have already checked

PHP version is the same

CircleCI

PHP 7.3.14 (cli) (built: Feb  1 2020 20:03:49) ( NTS )
Copyright (c) 1997-2018 The PHP Group
Zend Engine v3.3.14, Copyright (c) 1998-2018 Zend Technologies

Dev environment

PHP 7.3.14-6+ubuntu18.04.1+deb.sury.org+1 (cli) (built: Feb  5 2020 16:51:32) ( NTS )
Copyright (c) 1997-2018 The PHP Group
Zend Engine v3.3.14, Copyright (c) 1998-2018 Zend Technologies
    with Zend OPcache v7.3.14-6+ubuntu18.04.1+deb.sury.org+1, Copyright (c) 1999-2018, by Zend Technologies

PSALM version is the same

CircleCI

~/.composer/vendor/bin/psalm --version
Psalm dev-master@ffd04132ce1e40ae490ad2c43381cf4114893e04

Dev environment

~/.composer/vendor/bin/psalm --version
Psalm dev-master@ffd04132ce1e40ae490ad2c43381cf4114893e04

PSALM is using the same baseline file

PSALM is using the same config file

Running with --no-cache outputs the same results

Tested with version 3.8.3, same issue.

Tested with dev-master using phpVersion="7.3", same issue

What else should I check? I am really lost now.

---

Solved

Local environment and circleci's vendor folder were out of sync.

[orklah]

Do you have the same issue on last stable?

On master, there was a feature #2715 that reads composer file to set php version. If you switched to global install, maybe it doesn't read the same value on both?

[fkupper]

@orklah Thanks for your time.

• tested with 3.8.3, same issue.
• tested with dev-master using phpVersion, same issue

Updating the issue description with this extra infos.

[weirdan]

@fkupper are those errors that CircleCI reports suppressed via the baseline?

[fkupper]

@weirdan No. Just some more info:

I have updated my baseline to run with version 3.8.3 of psalm and tried again homestead env VS CircleCI and the same happened. Just to be more specific, running psalm with the same arguments but providing one specific file that is generating the divergent output, I get this:

Homestead

$ ~/.config/composer/vendor/bin/psalm --config=psalm.xml --show-info=false --use-ini-defaults --find-dead-code --no-cache app/Models/VatNumber.php
Scanning files...
Analyzing files...

░

To whom it may concern: Psalm cannot detect unused classes, methods and properties
when analyzing individual files and folders. Run on the full project to enable
complete unused code detection.

------------------------------
No errors found!
------------------------------
5 other issues found.
------------------------------
Psalm can automatically fix 2 of them.
Run Psalm again with 
--alter --issues=MissingReturnType --dry-run
to see what it can fix.
------------------------------

Checks took 10.76 seconds and used 165.765MB of memory
Psalm was able to infer types for 84.2697% of the codebase

CircleCI

$ /home/circleci/.composer/vendor/bin/psalm --config=psalm.xml --show-info=false --use-ini-defaults --find-dead-code --no-cache app/Models/VatNumber.php
Scanning files...
Analyzing files...

░

To whom it may concern: Psalm cannot detect unused classes, methods and properties
when analyzing individual files and folders. Run on the full project to enable
complete unused code detection.

ERROR: UndefinedMethod - app/Models/VatNumber.php:66:32 - Method Stripe\StripeObject::save does not exist
                $subscription->save();


------------------------------
1 errors found
------------------------------
5 other issues found.
------------------------------
Psalm can automatically fix 2 of them.
Run Psalm again with
--alter --issues=MissingReturnType --dry-run
to see what it can fix.
------------------------------

Checks took 3.35 seconds and used 165.271MB of memory
Psalm was able to infer types for 89.4737% of the codebase

So it is interesting to see that they both report the same count of other issues found and that the final line Psalm was able to infer types for % of the codebase on both have a different percentage.

Finally, if I run both again but with --show-info=true, the only difference is the UndefinedMethod found on CircleCI.

So my conclusion is that it has nothing to do with the baseline.

[weirdan]

Do vendor folders have the same contents in both cases?

[fkupper]

@weirdan After melting my mind with this, I have just deleted everything and spun up my environment again and indeed, some libraries were on slightly different versions. composer install ing everything from scratch made everything be synchronous again.

I think that running composer remove <library> for all the static analysis tools that I had as dev dependencies made my local app have different library versions from the fresh installs from CircleCI for some reason.

Thanks to everyone and sorry if I have wasted your time anyway.