forc index deploy --replace-indexer not working as expected

[ra0x3]

• Please see this Slack thread for context
  • We confirmed that the error happened on http://beta-4.fuel.network as well
    • This means that --replace-indexer is not actually replacing the indexer with a new indexer. How do I know?
      • First I removed the old indexer with forc index remove
      • Then I redeployed the explorer starting at block 7,000,000 and it works - no deserialization error
• This all leads me to believe that for whatever reason - --replace-indexer is not actually replacing the indexer properly
• I was not able to reproduce this locally - the explorer worked from block 7,000,000 with no issue
  • However, locally I did not try to use --replace-indexer
  • If you really wanted to repro you could:
    • Deploy an indexer using the last v0.23 release we had -- locally, on beta-4, starting from block 7,000,000
    • Let it index a few thousand blocks
    • Redeploy that indexer with v0.24.1 (without stopping the other one) and using --replace-data)
• Expected results
  • I would expect that the old indexer bytes get removed, and the new indexer bytes get registered - and the indexer executor task gets re-created, and indexing continues smoothly
    • The fact that I had to forc index remove before forc index deploy would work tells me the old indexer bytes must not have been removed 🤔

[ra0x3]

cc @lostman

[lostman]

@ra0x3, did you deploy with --replace-indexer and --remove-data?

If you did forc index remove, it would remove the indexer, and it's data.

However, forc index deploy --replace-indexer would only remove the indexer, not the data. So, if something and the serialized data stored in the database were no longer compatible, we'd get the deserialization error. I think the error was coming from get_object, so it would make sense.

We have a check that the deployed indexer has to be built with tooling with the same version. However, I don't think this applies to an already deployed indexer. Perhaps we could at least print something like:

WARN Indexer(some_indexer) was compiled with toolchain version A.B.C while the indexer service version is A.B.C. This may cause errors.

[ra0x3]

@lostman

• I did not use --remove-data because I did not want to remove the data
• I was forced to use forc index remove because forc index deploy --replace-indexer was simply (for whatever reason) not working
• I do not believe the QA env is running the indexer version check

Containers:
  fuel-indexer:
    Container ID:  containerd://1a3be8b56d343c06090867a99cb391026c94836cabfc2693f1f5f83777e22c8d
    Image:         ghcr.io/fuellabs/fuel-indexer:sha-118b6bd
    Image ID:      ghcr.io/fuellabs/fuel-indexer@sha256:9d1fbc97ea109c345db69fa91a6d855483bfec63cfba157ed6047c4735a1b118
    Port:          29987/TCP
    Host Port:     0/TCP
    Command:
      ./fuel-indexer
    Args:
      run
      --fuel-node-host
      beta-4.fuel.network
      --fuel-node-port
      80
      --postgres-password
      wlxwRcpcxiLpJaNKjJafpeLvIzQHvB
      --postgres-host
      indexer-dev-001.c9ddshmlrizp.us-east-1.rds.amazonaws.com
      --postgres-port
      5432
      --postgres-user
      postgres
      --postgres-database
      postgres
      --web-api-host
      0.0.0.0
      --run-migrations
      --auth-enabled
      --auth-strategy
      jwt
      --jwt-secret
      UiKWKUCzrECfwCXnOSJdnOPeuUFHbXRyCeixCweYkptEFHyryJ
      --jwt-issuer
      FuelLabs
      --rate-limit
      --rate-limit-request-count
      3
      --rate-limit-window-size
      10
      --verbose
      --replace-indexer
      --accept-sql-queries
      --block-page-size

• We should update this in a PR

[ra0x3]

@lostman Also keep in mind that --disable-toolchain-version-check is disabled by default, so if a version mismatch was the issue we would've saw that error when I was deploying