Exception in cci task upload_production - Malformed Request Error: The version number must be greater than the last Managed - Released version number

[kayla-blackthorn]

Describe the bug

I am encountering this exception when running the upload_production task as part of a GitHub Actions CumulusCI Workflow:

Malformed Request Error: Response content: ['message': 'The version number must be greater than the last Managed - Released version number: 1.16', 'errorCode':FIELD_INTEGRITY_EXCEPTION', 'fields': []]

I have a CCI workflow that runs using GitHub actions and for some reason a couple of the packages that use this workflow occasionally receive this error. If I run the CCI command manually after attempting to run the workflow, it correctly increments the version number and does not throw an error. I have the same automation in place for multiple packages and only experience this for some. There are no notable differences in the workflow files or cumulusci.yml files.

This action is run manually to create new releases. It deploys the latest code from the packaging org then proceeds to run a CCI flow that has upload_production as its first task. The packages have new major versions released on a monthly cadence. Of course, if there are tests that fail or bugs identified as part of our QA release testing/regression process, then we end up creating a new release candidate and running the action again. Most of these workflow runs are happening a month apart still though. For example, one package where I experienced the issue had its last version created 12/20/23 and I tried to run the action today 1/23/24. When I ran the flow manually (cci flow run release_production --org package-org) it worked fine without errors.

Reproduction steps

1. Kick off Salesforce Package Build workflow
2. When running the step to Create Major Release notice that the build fails for a Malformed Request Error on task upload_production

Your CumulusCI and Python versions

CumulusCI version: 3.77.0
Python version: 3.10.4

Operating System

macOS Ventura 13.4.1 (c)

Windows environment

No response

CumulusCI installation method

pip

Error Gist

No response

Additional information

No response

[jstvz]

@kayla-blackthorn We can't reproduce this issue using your reproduction steps, do you have any logs from the workflow run or the packaging org you could share? What about cci versions? Are they the same locally and in the action?

[kayla-blackthorn]

@jstvz Here is the most recent log: https://github.com/blackthornio/payments/actions/runs/7640963906/job/20817192501#step:10:64

My workflow runs pip install cumulusci, which I think installs the latest cci version?

I do see that in the logs there is a notice about a pip upgrade available:
Notice: A new release of pip is available: 23.0.1 -> 23.3.2
Notice: To update, run: pip install --upgrade pip

[kayla-blackthorn]

Hi just following up here

[jlantz]

That log is in a private repository that no one can see :) Could you copy/paste the relevant lines here?

[kayla-blackthorn]

@jlantz haha, sorry about that.

Error: Malformed request
https://blackthorn-payments-pac-dev-ed.my.salesforce.com/services/data/v48.0/too
ling/sobjects/PackageUploadRequest/. Response content: [{'message': 'The version
number must be greater than the last Managed - Released version number: 6.11',
'errorCode': 'FIELD_INTEGRITY_EXCEPTION', 'fields': []}]
Run this command for more information about debugging errors: cci error --help

Workflow:

name: Salesforce Package Build

on:
workflow_dispatch: {} # Enable manual triggering

env:
CUMULUSCI_KEYCHAIN_CLASS: cumulusci.core.keychain.EnvironmentProjectKeychain
CUMULUSCI_SERVICE_github: ${{ secrets.CUMULUSCI_SERVICE_GITHUB }}
SFDX_AUTH_URL: ${{ secrets.PACKAGING_ORG_AUTH_URL }}
SFDX_CLIENT_ID: ${{ secrets.SFDX_CLIENT_ID }}
SFDX_HUB_KEY: ${{ secrets.SFDX_HUB_KEY }}
CUMULUSCI_ORG_packaging: ${{ secrets.CUMULUSCI_ORG_PACKAGING }}

jobs:
build:
runs-on: ubuntu-latest

steps:
  - name: Check out repository
    uses: actions/checkout@v2

  - name: Set up Python
    uses: actions/setup-python@v2
    with:
      python-version: 3.8

  - name: Install CumulusCI
    run: pip install cumulusci
  
  - name: Install sfdx
    run: |
      mkdir sfdx
      wget -qO- https://developer.salesforce.com/media/salesforce-cli/sfdx/channels/stable/sfdx-linux-x64.tar.xz | tar xJ -C sfdx --strip-components 1
      export PATH="$(pwd)/sfdx/bin"
      echo "$(pwd)/sfdx/bin" >> $GITHUB_PATH

  - name: Convert to Metadata API Format
    run: |
      cci task run dx_convert_from

  - name: Update Package XML
    run: |
      cci task run update_package_xml --package_name "Blackthorn Base"

  - name: Deploy to Package Org
    run: |
      cci task run deploy --org packaging --unmanaged false

  - name: Determine Release Type
    id: release-type
    run: |
      if [[ "${{ vars.BETA_RELEASE }}" == "true" ]]; then
        echo "::set-output name=release-type::beta"
      else
        echo "::set-output name=release-type::major"
      fi
    continue-on-error: true

  - name: Create Beta Release
    if: steps.release-type.outputs.release-type == 'beta'
    run: |
      cci flow run release_beta --org packaging         

  - name: Create Major Release
    if: steps.release-type.outputs.release-type == 'major'
    run: |
      cci flow run release_production --org packaging

This is happening on Create Major Release, the first step of the custom flow release_production which is the upload_production task.

[kayla-blackthorn]

@jlantz I opened a case with Salesforce too and am waiting to hear. I want to make sure I understand how the upload_production task should be working. When I execute cci task run upload_production it automatically increments to the next minor version. When it gets executed via the workflow it looks like it sometimes tries to use the current minor version.

[kayla-blackthorn]

I just finished reviewing with Salesforce support, and of course the Action I ran on the call succeeded to increment the version number. Even before running through a demo for them, Salesforce was adamant that the issue was not on their end. They said that error would only be thrown if the version attempting to be said was not ahead of the latest package version. I double checked to make sure that the latest release was marked correctly in GitHub since I now know that the cumulus flows look at the github releases to determine the latest. Those are however marked correctly in two of my repos where I just ran my automation. One worked (on the demo to Salesforce) while the other failed due to [{'message': 'The version metadata currently in packaging number must be greater than the last Managed - Released version number: 6.13', 'errorCode': 'FIELD_INTEGRITY_EXCEPTION', 'fields': []}]

[kayla-blackthorn]

I started trying to better understand exactly what cci is doing with the upload_production task. If I am understanding package_upload.py correctly, it looks like I might see why the version error is happening.

When I run task upload_production I am not explicitly setting major_version or minor_version in options. So it looks like it fetches the latest version from the database using this query: SELECT MajorVersion, MinorVersion, PatchVersion, ReleaseState FROM MetadataPackageVersion ORDER BY MajorVersion DESC, MinorVersion DESC, PatchVersion DESC, ReleaseState DESC LIMIT 1
We always create a patch version after each release before creating the next release version just in case we need it. So, that query returns MetadataPackageVersion:{MajorVersion=6, MinorVersion=13, PatchVersion=1, ReleaseState=Beta, Id=04tKY000000TpBdYAK}
So I would think that it sets self.options["major_version"] to "6" since it's not explicitly set in the options and thats the latest major version returned. Since self.options["major_version"] is then equal to the latest major version from the org, it proceeds to check for the presence of minor_version. Since minor_version is not present in the options, it sets self.options["minor_version"] to "0" because the release state is "Beta".

My package that is NOT experiencing this issue is one that has not yet passed security review and thus is not eligible for patch orgs.

If I am correct in interpreting how this is working, I assume I cannot actually use the upload_production task as is and will need to write some custom logic to set the version correctly so that it increments to (in this case) 6.14

[kayla-blackthorn]

I guess even explicitly setting the version numbers in options won't work if the latest version in the org is a Beta. If I try to set 6.14, I will hit the error: Latest Minor Version is Beta so minor version cannot be greater than that.

[kayla-blackthorn]

On our end, we will hold on creating patch versions until after the next monthly release version has been created to avoid that query returning the beta patch version. Is it possible though to have it updated to filter out deprecated versions? I was going to test to confirm that as long as the version returned by the query was the latest non-beta version that it would work as expected, but deprecating my patch version won't change anything since the query still finds it as the latest version.