Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Error The metadata storage is not up to date since 3.5.3 #1302

Closed
dmaicher opened this issue Jan 11, 2023 · 27 comments
Closed

Error The metadata storage is not up to date since 3.5.3 #1302

dmaicher opened this issue Jan 11, 2023 · 27 comments

Comments

@dmaicher
Copy link
Contributor

dmaicher commented Jan 11, 2023

Bug Report

Q A
BC Break yes?
Version 3.5.3

Summary

I just upgraded to 3.5.3 and I have an error when running app/console doctrine:migrations:up-to-date:

In MetadataStorageError.php line 13:
                                                                                                           
  The metadata storage is not up to date, please run the sync-metadata-storage command to fix this issue.                                                                                                   

Unfortunately running doctrine:migrations:sync-metadata-storage does not help. It says [OK] Metadata storage synchronized but I get the same error when running up-to-date again.

How to reproduce

  • Mysql 5.7.40
  • doctrine/dbal 3.5.2
  • doctrine/doctrine-bundle 2.8.2
  • doctrine/doctrine-migrations-bundle 3.2.2
  • Symfony 6.2

This is my metadata table dump

CREATE TABLE `migration_versions` (
  `version` varchar(191) COLLATE utf8_unicode_ci NOT NULL,
  `executed_at` datetime DEFAULT NULL,
  `execution_time` int(11) DEFAULT NULL,
  PRIMARY KEY (`version`)
) ENGINE=InnoDB DEFAULT CHARSET=utf8 COLLATE=utf8_unicode_ci;

Expected behavior

It should not report an issue with out-of-sync storage

@ochov1
Copy link

ochov1 commented Jan 11, 2023

Here the same. php 7.4 symfony 5.4

@ochov1
Copy link

ochov1 commented Jan 11, 2023

My solution was to downgrade to 5.3.2

composer require doctrine/migrations:3.5.2

@dmaicher
Copy link
Contributor Author

This is the table diff that is reported here:

^ Doctrine\DBAL\Schema\TableDiff^ {#2304
  +name: "migration_versions"
  +newName: false
  +addedColumns: []
  +changedColumns: array:1 [
    "version" => Doctrine\DBAL\Schema\ColumnDiff^ {#2313
      +oldColumnName: "version"
      +column: Doctrine\DBAL\Schema\Column^ {#2320
        #_name: "version"
        #_namespace: null
        #_quoted: false
        #_type: Doctrine\DBAL\Types\StringType^ {#2446}
        #_length: 191
        #_precision: 10
        #_scale: 0
        #_unsigned: false
        #_fixed: false
        #_notnull: true
        #_default: null
        #_autoincrement: false
        #_platformOptions: []
        #_columnDefinition: null
        #_comment: null
        #_customSchemaOptions: []
      }
      +changedProperties: []
      +fromColumn: Doctrine\DBAL\Schema\Column^ {#2280
        #_name: "version"
        #_namespace: null
        #_quoted: false
        #_type: Doctrine\DBAL\Types\StringType^ {#2446}
        #_length: 191
        #_precision: 10
        #_scale: 0
        #_unsigned: false
        #_fixed: false
        #_notnull: true
        #_default: null
        #_autoincrement: false
        #_platformOptions: array:2 [
          "charset" => "utf8"
          "collation" => "utf8_unicode_ci"
        ]
        #_columnDefinition: null
        #_comment: null
        #_customSchemaOptions: []
      }
    }
  ]
  +removedColumns: []
  +renamedColumns: []
  +addedIndexes: []
  +changedIndexes: []
  +removedIndexes: []
  +renamedIndexes: []
  +addedForeignKeys: []
  +changedForeignKeys: []
  +removedForeignKeys: []
  +fromTable: Doctrine\DBAL\Schema\Table^ {#2330
    #_name: "migration_versions"
    #_namespace: null
    #_quoted: false
    #_columns: array:3 [
      "version" => Doctrine\DBAL\Schema\Column^ {#2280}
      "executed_at" => Doctrine\DBAL\Schema\Column^ {#2340
        #_name: "executed_at"
        #_namespace: null
        #_quoted: false
        #_type: Doctrine\DBAL\Types\DateTimeType^ {#2481}
        #_length: 0
        #_precision: 10
        #_scale: 0
        #_unsigned: false
        #_fixed: false
        #_notnull: false
        #_default: null
        #_autoincrement: false
        #_platformOptions: []
        #_columnDefinition: null
        #_comment: null
        #_customSchemaOptions: []
      }
      "execution_time" => Doctrine\DBAL\Schema\Column^ {#2342
        #_name: "execution_time"
        #_namespace: null
        #_quoted: false
        #_type: Doctrine\DBAL\Types\IntegerType^ {#2451}
        #_length: null
        #_precision: 10
        #_scale: 0
        #_unsigned: false
        #_fixed: false
        #_notnull: false
        #_default: null
        #_autoincrement: false
        #_platformOptions: []
        #_columnDefinition: null
        #_comment: null
        #_customSchemaOptions: []
      }
    ]
    #_indexes: array:1 [
      "primary" => Doctrine\DBAL\Schema\Index^ {#2316
        #_name: "PRIMARY"
        #_namespace: null
        #_quoted: false
        #_columns: array:1 [
          "version" => Doctrine\DBAL\Schema\Identifier^ {#2311
            #_name: "version"
            #_namespace: null
            #_quoted: false
          }
        ]
        #_isUnique: true
        #_isPrimary: true
        #_flags: []
        -options: array:1 [
          "lengths" => array:1 [
            0 => null
          ]
        ]
      }
    ]
    #_primaryKeyName: "primary"
    #uniqueConstraints: []
    #_fkConstraints: []
    #_options: array:6 [
      "create_options" => []
      "engine" => "InnoDB"
      "collation" => "utf8_unicode_ci"
      "charset" => "utf8"
      "autoincrement" => null
      "comment" => ""
    ]
    #_schemaConfig: null
    -implicitIndexes: []
  }
}

@colinpieper
Copy link

Our Symfony 5.4 projects are also impacted by this issue, running MariaDB 10.6.9. Downgrading doctrine/migrations to 3.5.2 fixed the issue.

@yani
Copy link

yani commented Jan 11, 2023

Same happened over here.

I'm using the library natively with MariaDB and changing server_version to mariadb-x.x.x did not fix the issue.

Downgrading to 3.5.2 fixed it.

@Kaaly
Copy link

Kaaly commented Jan 11, 2023

Same issue here.
Seem coming from this commit : 315f255aa78c1179197759a6afa9608437ea2ef6

@axi
Copy link

axi commented Jan 11, 2023

The difference is on the "_platformOptions" for me.
On database ($fromTable):

[
       "charset" => "utf8mb4"
       "collation" => "utf8mb4_unicode_ci"
]

On $toTable:

[]

I would say introduced in this commit 1f58518

Reverting method needsUpdate() from TableMetadataStorage fix the problem on my side

    private function needsUpdate(Table $expectedTable): ?TableDiff
    {
        if ($this->schemaUpToDate) {
            return null;
        }
        $currentTable = $this->schemaManager->listTableDetails($this->configuration->getTableName());
        $diff         = $this->schemaManager->createComparator()->diffTable($currentTable, $expectedTable);

        return $diff instanceof TableDiff ? $diff : null;
    }

@WoutervanderLoopNL
Copy link

Same here, not getting the charset en collation. so its see a diff every time

@greg0ire
Copy link
Member

@axi (or anybody) there are 3 changes in that method, it might be interesting to revert only one of them.

@greg0ire
Copy link
Member

Also @axi, Table does not have a _platformOptions property, does it? Only Column does: https://github.com/doctrine/dbal/blob/e106ee7a3869a2ca82b0993914ea55061be02554/src/Schema/Column.php#L45-L46

On which column(s) do you observe the diff?

@stof
Copy link
Member

stof commented Jan 11, 2023

@greg0ire in the dump of the TableDiff in @dmaicher's comment above, the column appearing in changedColumns is the version column.

@greg0ire
Copy link
Member

greg0ire commented Jan 11, 2023

I think something must be calling setPlatformOptions or setPlatformOption on that column with these options at some point, and maybe the same should be done here somehow:

private function getExpectedTable(): Table
{
$schemaChangelog = new Table($this->configuration->getTableName());
$schemaChangelog->addColumn(
$this->configuration->getVersionColumnName(),
'string',
['notnull' => true, 'length' => $this->configuration->getVersionColumnLength()]
);
$schemaChangelog->addColumn($this->configuration->getExecutedAtColumnName(), 'datetime', ['notnull' => false]);
$schemaChangelog->addColumn($this->configuration->getExecutionTimeColumnName(), 'integer', ['notnull' => false]);
$schemaChangelog->setPrimaryKey([$this->configuration->getVersionColumnName()]);
return $schemaChangelog;
}

Can one of you determine if that's the case? I'm suspecting https://github.com/doctrine/dbal/blob/3.5.x/src/Platforms/MySQL/Comparator.php#L62

What's very interesting is the comment on that class: https://github.com/doctrine/dbal/blob/e106ee7a3869a2ca82b0993914ea55061be02554/src/Platforms/MySQL/Comparator.php#L15-L17

In MySQL, unless specified explicitly, the column's character set and collation are inherited from its containing
table. So during comparison, an omitted value and the value that matches the default value of table in the
desired schema must be considered equal.

If I understand correctly, the diff should be empty if utf8mb4 and utf8mb4_unicode_ci is what is used at the table level.

@dmaicher
Copy link
Contributor Author

dmaicher commented Jan 11, 2023

nice catch @greg0ire . As discussed on Slack the issue was the missing normalization of columns: doctrine/dbal#5861

@SchwarzDeveloping
Copy link

Same here. Symfony 6.2, PHP 8.1, Mariadb 10.5.15
Downgrade to 3.5.2 fixed it for the moment.
Took 10 hours, 5 tantrums and 8 project restarts to find this issue -.-

@greg0ire
Copy link
Member

Please stop with the spam.

@greg0ire
Copy link
Member

greg0ire commented Jan 12, 2023

Fixed by https://github.com/doctrine/dbal/releases/tag/3.5.3 (don't get confused, it's another package, it just happens to have the same version number)

@samuel-sol
Copy link

samuel-sol commented Jan 12, 2023

@greg0ire I've tried updating again with composer update but the error persist. Forcing a downgrade to 3.5.2 and then reupgrading do 3.5.3 didn't work. removing the package and readding also did not fix.

Here is a var_dump part of the return from the method Doctrine\Migrations\Metadata\Storage\TableMetadataStorage::needsUpdate. You can see that the error is that for some reason he has "default => (string)NULL" instead of NULL.

object(Doctrine\DBAL\Schema\TableDiff)#1005 (14) {
  ["changedColumns"]=>
  array(1) {
    ["executed_at"]=>
    object(Doctrine\DBAL\Schema\ColumnDiff)#1004 (4) {
      ["oldColumnName"]=>
      string(11) "executed_at"
      ["column"]=>
      object(Doctrine\DBAL\Schema\Column)#1001 (16) {
        ["_name":protected]=>
        string(11) "executed_at"
        ["_namespace":protected]=>
        NULL
        ["_quoted":protected]=>
        bool(false)
        ["_type":protected]=>
        object(Doctrine\DBAL\Types\DateTimeType)#946 (0) {
        }
        ["_length":protected]=>
        NULL
        ["_precision":protected]=>
        int(10)
        ["_scale":protected]=>
        int(0)
        ["_unsigned":protected]=>
        bool(false)
        ["_fixed":protected]=>
        bool(false)
        ["_notnull":protected]=>
        bool(false)
        ["_default":protected]=>
        NULL
        ["_autoincrement":protected]=>
        bool(false)
        ["_platformOptions":protected]=>
        array(0) {
        }
        ["_columnDefinition":protected]=>
        NULL
        ["_comment":protected]=>
        NULL
        ["_customSchemaOptions":protected]=>
        array(0) {
        }
      }
      ["changedProperties"]=>
      array(1) {
        [0]=>
        string(7) "default"
      }
      ["fromColumn"]=>
      object(Doctrine\DBAL\Schema\Column)#996 (16) {
        ["_name":protected]=>
        string(11) "executed_at"
        ["_namespace":protected]=>
        NULL
        ["_quoted":protected]=>
        bool(false)
        ["_type":protected]=>
        object(Doctrine\DBAL\Types\DateTimeType)#946 (0) {
        }
        ["_length":protected]=>
        int(0)
        ["_precision":protected]=>
        int(10)
        ["_scale":protected]=>
        int(0)
        ["_unsigned":protected]=>
        bool(false)
        ["_fixed":protected]=>
        bool(false)
        ["_notnull":protected]=>
        bool(false)
        ["_default":protected]=>
        string(4) "NULL"
        ["_autoincrement":protected]=>
        bool(false)
        ["_platformOptions":protected]=>
        array(0) {
        }
        ["_columnDefinition":protected]=>
        NULL
        ["_comment":protected]=>
        NULL
        ["_customSchemaOptions":protected]=>
        array(0) {
        }
      }
    }
  }

and here is my composer.json

{
    "type": "project",
    "license": "proprietary",
    "minimum-stability": "stable",
    "prefer-stable": true,
    "require": {
        "php": ">=8.1.0",
        "ext-ctype": "*",
        "ext-iconv": "*",
        "ext-openssl": "*",
        "beberlei/doctrineextensions": "^1.3",
        "doctrine/annotations": "^1.13.3",
        "doctrine/doctrine-bundle": "^2.0",
        "doctrine/migrations": "^3.0",
        "doctrine/doctrine-migrations-bundle": "^3.0",
        "doctrine/dbal": "^3.0",
        "doctrine/orm": "2.14.0",
        "doctrine/persistence": "^2.5.0",
        "gedmo/doctrine-extensions": "^3.0.0",
        "knplabs/knp-menu-bundle": "^3.2",
        "laminas/laminas-code": "4.7.0",
        "phpdocumentor/reflection-docblock": "^5.3",
        "phpstan/phpdoc-parser": "^1.6.4",
        "stof/doctrine-extensions-bundle": "*",
        "symfony/asset": "6.*",
        "symfony/console": "6.*",
        "symfony/doctrine-messenger": "6.*",
        "symfony/dotenv": "6.*",
        "symfony/expression-language": "6.*",
        "symfony/flex": "^2.2",
        "symfony/form": "6.*",
        "symfony/framework-bundle": "6.*",
        "symfony/http-client": "6.*",
        "symfony/intl": "6.*",
        "symfony/ldap": "6.*",
        "symfony/mailer": "6.*",
        "symfony/maker-bundle": "^1.45",
        "symfony/mime": "6.*",
        "symfony/monolog-bundle": "^3.0",
        "symfony/notifier": "6.*",
        "symfony/process": "6.*",
        "symfony/property-access": "6.*",
        "symfony/property-info": "6.*",
        "symfony/proxy-manager-bridge": "6.*",
        "symfony/runtime": "6.*",
        "symfony/security-bundle": "6.*",
        "symfony/serializer": "6.*",
        "symfony/string": "6.*",
        "symfony/translation": "6.*",
        "symfony/twig-bundle": "6.*",
        "symfony/validator": "6.*",
        "symfony/web-link": "6.*",
        "symfony/webapp-meta": "^1.0",
        "symfony/webpack-encore-bundle": "^1.15",
        "symfony/yaml": "6.*",
        "twig/cssinliner-extra": "^3.3",
        "twig/extra-bundle": "^3.4",
        "twig/inky-extra": "^3.3",
        "twig/intl-extra": "^3.4",
        "twig/twig": "^3.0"
    },
    "config": {
        "allow-plugins": {
            "composer/package-versions-deprecated": true,
            "symfony/flex": true,
            "symfony/runtime": true
        },
        "optimize-autoloader": true,
        "preferred-install": {
            "*": "dist"
        },
        "sort-packages": true
    },
    "autoload": {
        "psr-4": {
            "App\\": "src/"
        }
    },
    "autoload-dev": {
        "psr-4": {
            "App\\Tests\\": "tests/"
        }
    },
    "replace": {
        "symfony/polyfill-ctype": "*",
        "symfony/polyfill-iconv": "*",
        "symfony/polyfill-php72": "*",
        "symfony/polyfill-php73": "*",
        "symfony/polyfill-php74": "*",
        "symfony/polyfill-php80": "*"
    },
    "scripts": {
        "auto-scripts": {
            "assets:install %PUBLIC_DIR%": "symfony-cmd"
        },
        "post-install-cmd": [
            "@auto-scripts"
        ],
        "post-update-cmd": [
            "@auto-scripts"
        ]
    },
    "conflict": {
        "symfony/symfony": "*"
    },
    "extra": {
        "symfony": {
            "allow-contrib": false,
            "require": "6.2.*"
        }
    },
    "require-dev": {
        "symfony/debug-bundle": "6.*",
        "symfony/stopwatch": "6.*",
        "symfony/var-dumper": "6.*",
        "symfony/web-profiler-bundle": "6.*"
    },
    "repositories": [
        {
            "type": "vcs",
            "url": "spoecial"
        },
        {
            "type": "path",
            "url": local"
        }
    ]
}

@samuel-sol
Copy link

Ok, removing from require-dev the symfony/web-profiler-bundle removes the issue, even though this package was not updated recently.

@greg0ire
Copy link
Member

Lol what

@greg0ire
Copy link
Member

@samuel-sol reading your first message, it sounds like you downgraded doctrine/migrations to 2.5.2, then upgraded again to 2.5.3, expecting the bug to be somehow fixed… if that's the case, I suggest you read #1302 (comment) again, but slowly.

@TTATPuOT
Copy link

TTATPuOT commented Jan 23, 2023

Same problem here with MySQL 8.0.25, PHP 7.4. But im use old Doctrine 2.8.0.

@spajxo
Copy link

spajxo commented Jun 20, 2023

If you are using getsentry/sentry-symfony this could be related
getsentry/sentry-symfony#732

@greg0ire greg0ire unpinned this issue Sep 2, 2023
@MichaelMackus
Copy link

Same problem for me .... I'm on doctrine/migrations 3.6.0

@kwisatz
Copy link

kwisatz commented May 1, 2024

EDIT: I realize that this is an old issue, but this just came up on my project and I cannot see in the comments above, a clear indication of what is causing this or how to go about finding the correct solution.

Fixed by https://github.com/doctrine/dbal/releases/tag/3.5.3 (don't get confused, it's another package, it just happens to have the same version number)

@greg0ire This does not seem to solve the problem in all cases.
I am running dbal 3.8.4 and migrations 3.5.5 with the changes shown in https://github.com/doctrine/dbal/pull/5861/files, yet, the problem is still there or another problem with the same symptoms is. I have the full mariadb version string indicated in my DATABASE_URL environment variable (?serverVersion=mariadb-10.11.6&charset=utf8).

Here's a dump of the TableDiff object. As @samuel-sol mentioned, it seems to want to change null to "NULL" for the column defaults:

^ Doctrine\DBAL\Schema\TableDiff^ {#477
  +name: "migration_versions"
  +newName: false
  +addedColumns: []
  +changedColumns: array:1 [
    "executed_at" => Doctrine\DBAL\Schema\ColumnDiff^ {#488
      +oldColumnName: "executed_at"
      +column: Doctrine\DBAL\Schema\Column^ {#495
        #_name: "executed_at"
        #_namespace: null
        #_quoted: false
        #_type: Doctrine\DBAL\Types\DateTimeType^ {#515}
        #_length: null
        #_precision: 10
        #_scale: 0
        #_unsigned: false
        #_fixed: false
        #_notnull: false
        #_default: null
        #_autoincrement: false
        #_platformOptions: []
        #_columnDefinition: null
        #_comment: null
        #_customSchemaOptions: []
      }
      +changedProperties: array:1 [
        0 => "default"
      ]
      +fromColumn: Doctrine\DBAL\Schema\Column^ {#534
        #_name: "executed_at"
        #_namespace: null
        #_quoted: false
        #_type: Doctrine\DBAL\Types\DateTimeType^ {#515}
        #_length: 0
        #_precision: 10
        #_scale: 0
        #_unsigned: false
        #_fixed: false
        #_notnull: false
        #_default: "NULL"
        #_autoincrement: false
        #_platformOptions: []
        #_columnDefinition: null
        #_comment: null
        #_customSchemaOptions: []
      }
    }
  ]
  +removedColumns: []
  +renamedColumns: []
  +addedIndexes: []
  +changedIndexes: []
  +removedIndexes: []
  +renamedIndexes: []
  +addedForeignKeys: []
  +changedForeignKeys: []
  +removedForeignKeys: []
  +fromTable: Doctrine\DBAL\Schema\Table^ {#533
    #_name: "migration_versions"
    #_namespace: null
    #_quoted: false
    #_columns: array:3 [
      "version" => Doctrine\DBAL\Schema\Column^ {#528
        #_name: "version"
        #_namespace: null
        #_quoted: false
        #_type: Doctrine\DBAL\Types\StringType^ {#505}
        #_length: 191
        #_precision: 10
        #_scale: 0
        #_unsigned: false
        #_fixed: false
        #_notnull: true
        #_default: null
        #_autoincrement: false
        #_platformOptions: []
        #_columnDefinition: null
        #_comment: null
        #_customSchemaOptions: []
      }
      "executed_at" => Doctrine\DBAL\Schema\Column^ {#534}
      "execution_time" => Doctrine\DBAL\Schema\Column^ {#535
        #_name: "execution_time"
        #_namespace: null
        #_quoted: false
        #_type: Doctrine\DBAL\Types\IntegerType^ {#538}
        #_length: null
        #_precision: 10
        #_scale: 0
        #_unsigned: false
        #_fixed: false
        #_notnull: false
        #_default: "NULL"
        #_autoincrement: false
        #_platformOptions: []
        #_columnDefinition: null
        #_comment: null
        #_customSchemaOptions: []
      }
    ]
    #_indexes: array:1 [
      "primary" => Doctrine\DBAL\Schema\Index^ {#536
        #_name: "PRIMARY"
        #_namespace: null
        #_quoted: false
        #_columns: array:1 [
          "version" => Doctrine\DBAL\Schema\Identifier^ {#525
            #_name: "version"
            #_namespace: null
            #_quoted: false
          }
        ]
        #_isUnique: true
        #_isPrimary: true
        #_flags: []
        -options: array:1 [
          "lengths" => array:1 [
            0 => null
          ]
        ]
      }
    ]
    #_primaryKeyName: "primary"
    #uniqueConstraints: []
    #_fkConstraints: []
    #_options: array:6 [
      "create_options" => []
      "engine" => "InnoDB"
      "collation" => "utf8mb3_unicode_ci"
      "charset" => "utf8mb3"
      "autoincrement" => null
      "comment" => ""
    ]
    #_schemaConfig: null
    -implicitIndexes: []
  }
}

@greg0ire
Copy link
Member

greg0ire commented May 1, 2024

Since OP seems to be satisfied with this, it means this is a different issue with the same symptoms. You should open another issue if there isn't one already about this null vs "NULL" thingy.

@kwisatz
Copy link

kwisatz commented May 1, 2024

@greg0ire I might. I have a feeling this is some incompatibility between some doctrine component and the mariadb version we're running. I've seen more instances of quoted NULL strings in the migration I was able to create by temporarily deactivating the needsUpdate check.

@greg0ire
Copy link
Member

greg0ire commented May 1, 2024

OK. In the meantime, I'll lock this, as the error message can be caused by many different reasons.

@doctrine doctrine locked as off-topic and limited conversation to collaborators May 1, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

No branches or pull requests