Improve addColumn handling in SQLite

[nvoxland]

Environment

Liquibase Version: 4.4.0
Database Vendor & Version: SQLite

Description

Fixed a variety of issues with SQLite in addColumn. These problems were found by existing liquibase-test-harness tests that are now passing.

Fixed handling of multi-column addColumn statements

Sqlite failed running liquibase update with an addColumn change that used multiple columns.

<addColumn tableName="authors">
            <column name="varcharColumn" type="varchar(25)" value="INITIAL_VALUE"/>
            <column name="intColumn" type="int" valueNumeric="5"/>
            <column name="dateColumn" type="date" valueDate="2020-09-21"/>
        </addColumn>

Repo Steps

1. Create a changeset using the above change
2. Run liquibase update

Without the fix, it did not correctly add all 3 columns.

Fixed onDelete cascade

The onDelete=cascade for argument was not respected in addColumn changes. And, if a table contains existing foreign keys with onDelete=cascade, adding columns to that table loses that setting.

Repo Steps:

• Fixes addColumn ignores deleteCascade #1671
• Fixes addColumn breaks existing foreign keys #1896

Fixed multi-column dropColumn

Sqlite failed running liquibase update with an dropColumn change that used multiple columns.

<dropColumn tableName="authors">
            <column name="varcharColumn"/>
            <column name="intColumn"/>
            <column name="dateColumn"/>
        </addColumn>

Repo Steps

1. Create a changeset using the above change
2. Run liquibase update

Without the fix, it failed with a "column does not exist" error.

ALTERNATE:

1. create a changeset with <addColumn> using multiple nested {{}}s.
2. Run liquibase update to apply it
3. Run liquibase rollbackCount 1 to roll it back

Without the fix, it failed with a "column does not exist" error.

JSON Snapshot Serializer should not serialize null values

Sqlite does not use a username, and liquibase snapshot --format json would include username: "null" which is not correct. Instead, null values should be not included.

Repo steps

1. Run liquibase snapshot --format json

The output should not include "null" for username or any attributes.

AddColumn fails with mixed-case column names

Fixed an issue with how quoting was used in the CopyRowsGenerator logic that is ran as part of addColumn. The multi-column example from test harness was using mixed-case column names which tripped up on the way quoting was done before.

Repo Steps

1. <createTable> with all objects quoted, and mixed casing on column names
2. insert data into that table
3. Create an <addColumn> to the table
4. Run liquibase update

Prior to the fix, the update failed

---

Dev Handoff Notes (Internal Use)

Links

• Fixed Issues:
  • addColumn ignores deleteCascade #1671
  • addColumn breaks existing foreign keys #1896
• Local Branch: https://github.com/liquibase/liquibase/tree/sqlite-improvements
• Unit Tests: https://github.com/liquibase/liquibase/pull/1970/checks
• Integration Tests: (shown in Unit Tests link above)
• Functional Tests: https://jenkins.datical.net/job/liquibase-pro/job/sqlite-improvements/

Testing

• Steps to Reproduce:
  • addColumn ignores deleteCascade #1671
  • addColumn breaks existing foreign keys #1896
  • Multi-column example above
• Guidance:
  • Fixes are generally isolated to sqlite but could improve other database support as well.
  • PR collects all the changes needed to make existing test-harness tests pass, did not try to fix anything beyond the tests that were failing and now passing.

Dev Verification

Ran a variety of addColumn and dropColumn tests locally, as well as ensured the test harness did not hit these problems anymore.

Test Requirements (Liquibase Internal QA)

Manual Tests

Add and Drop Multi-Column Changesets

Execute the manual tests against any of the Liquibase certified database platforms. After you deploy the XML changesets for creating the table and adding columns, you can use liquibase generate-changelog to create the JSON and YML changelogs. Of course, you do not want to try and generate those changelogs after you run the dropColumn changeset!

XML

<changeSet  id="1::table"  author="Liquibase Pro User" labels="table">  
    <createTable  tableName="authors">  
        <column  name="id"  type="int"/>  
    </createTable>  
</changeSet>

<changeSet  id="2::data"  author="Liquibase Pro User" labels="data">  
    <insert tableName="authors">  
        <column  name="id"  value="10"/>  
    </insert>  
    <insert tableName="authors">  
        <column  name="id"  value="20"/>  
    </insert>  
    <insert tableName="authors">  
        <column  name="id"  value="30"/>  
    </insert>  
</changeSet>

<changeSet  id="3::multiColumnAdd"  author="Liquibase Pro User" labels="addColumns">  
    <addColumn tableName="authors">
        <column name="varcharColumn" type="varchar(25)" value="INITIAL_VALUE"/>
        <column name="intColumn" type="int" valueNumeric="5"/>
        <column name="dateColumn" type="date" valueDate="2020-09-21"/>
    </addColumn>
</changeSet>

<changeSet  id="4::multiColumnDrop"  author="Liquibase Pro User" labels="dropColumns">  
    <dropColumn tableName="authors">
        <column name="varcharColumn"/>
        <column name="intColumn"/>
        <column name="dateColumn"/>
    </addColumn>
</changeSet>

Verify an update of a multi-column addColumn changeset is successful.

• liquibase update --changelog-file <yourchangelog.xml> --labels table,addColumns
• XML
• YML
• JSON

Verify an update of a multi-column dropColumn changeset is successful.

• liquibase update --changelog-file <yourchangelog.xml> --labels dropColumns
• XML
• YML
• JSON

---

Manual Test: Null Username in Snapshot

Setup

• This test case requires a SQLite database to duplicate the JDBC connection string that does not contain a username.
• Use the liquibase/liquibase-test-harness to launch a SQLite Docker.
  • Edit the https://github.com/liquibase/liquibase-test-harness/blob/main/src/test/resources/harness-config.yml to include only SQLite.
  • ⚠️ Do not commit this change back to source control! ⚠️
• Configure a liquibase.properties file with a connection to the SQLite DB.
  • Connection string: jdbc:sqlite:./src/test/resources/sqlite/sqlite.db

Verify that a JSON snapshot does not have “null” for the username.

• liquibase snapshot --format json

---

Manual Test: Casing

Verify adding columns to a table with mixed-case columns is successful.

• liquibase update --changelog-file <yourchangelog.xml> --labels table,data,addColumns
• XML
• YML
• JSON

Verify the data is present in the table after update is complete.

• Connect to the database and run select * authors

---

Automated Tests

• Add new functional automated test to the liquibase-pro-tests suite.
• Write a multi-db functional test that validates a multi-column addColumn and a multi-column dropColumn that:
  • validates update of addColumn is successful
  • validates rollback of addColumn is successful
  • validates update of dropColumn is successful
  • validates rollback of dropColumn is successful

┆Issue is synchronized with this Jira Bug by Unito

[KushnirykOleh]

can we expand this to other DBs? I found Firebird need same fix as it also fails with liquibase.exception.UnexpectedLiquibaseException: Found a null snapshotId for catalog null

[KushnirykOleh]

changing if condition to include (database instanceof SQLiteDatabase || database instanceof FirebirdDatabase) helps with Firebird snapshot generation, but maybe this can be expanded further

[kristyldatical]

@nvoxland -- please see @KushnirykOleh's comment above and provide guidance.

[nvoxland]

Yeah, it is probably safe to include for all database, not just sqlite and firebird. I updated the code to just be if (returnlist.size() == 0)