Significant performance degradation after migration from 3.5.3 to 4.26.0

[Fedoop1]

Search first

• I searched and no similar issues were found

Description

After liquibase migration from 3.5.3 to 4.26.0 we noticed that the total liquibase update time became very unstable.
First of all overall memory consumption increased and now there are a few spikes that vary from 4 GB to 5 GB per execution (most of them are byte arrays) after which process private memory doesn't shrink. On 3.5.3 we didn't have such a problem.

The second problem is when there is memory pressure and ram utilization is close to the end, the total execution time can be several times longer than when there is enough memory for the execution. Before migration, it was stable for 3 minutes, but now it can last for 7-9 minutes. I'd guess that it's due to the problem mentioned above and probably OS can't dedicate so much memory to the liquibase process.

Steps To Reproduce

Please find the archive attached. You can run a test.ps1 script and monitor performance. There is only one big migration without sensitive data. NB: on the screenshot, you can see the performance overview from JMC of test db creation with the attached transaction included. Unfortunately, I can't attach all migrations due to their sensitivity.
liquibase-test.zip

Expected/Desired Behavior

Predictable average memory consumption. Shrink after spike. Predictable performance (execution time)

Liquibase Version

4.26.0

Database Vendor & Version

SQL Server 13.0.5893.48

Liquibase Integration

CLI

Liquibase Extensions

No response

OS and/or Infrastructure Type/Provider

Windows 10 Enterprise

Additional Context

I tried to look into this problem and found a few migrations after which I usually see some spikes. They are huge bulk inserts.
I moved SQL code from .xml changelog files to .sql and group insert statements into batches, spikes reduced from 4-5 GB to 2.5-3 GB per average execution, but it's still very high.

Sometimes I see strange runs when process memory doesn't exceed 1-2 GB and can't explain it to myself.

Are you willing to submit a PR?

• I'm willing to submit a PR (Thank you!)

[kevin-atx]

@Fedoop1, thank you for your thorough investigation and writeup of this issue. I will add this issue to the next sprint for review. Someone on our team will reach out if they have questions or need to discuss this further.

[MalloD12]

Hi @Fedoop1,

I've been trying to reproduce this issue using the below environment:

• Liquibase 4.26, 4.27 and 4.28
• SQL Server 14.00.3465
• Window 10

And having the below heap size:

 size_t MaxHeapSize                              = 3049259008 

I tried executing the changelog you provided ChangeLog.xml just running liquibase udpate and it was executed only spending some seconds. I tried also executing the test.ps1 script and the output and executing time was pretty much the same.

Was your heap size set to 4GB? If not can you tell us what's the memory size you have been using? So I can set mine in the same way and try to reproduce it with similar characteristics.

On the other, I know you mentioned you can share some of the changelogs you are using because it contains sensitive data, but I was wondering if this performance degradation issue was more noticeable with one of those changelogs. If yes, could you please tell us what were the most change types it contains? That way myself or someone from the team can craft a changelog and try with it.

Thanks in advance,
Daniel.

[Fedoop1]

Hi @MalloD12,

Thanks for the feedback!

To avoid confusion: the screenshots I've attached before were from different changelog and liquibase-test.zip is just a changelog with one changeset from our internal changelog for test db creation.

Here is what I get when running test.ps1 locally on a fresh db:

With the heap size size_t MaxHeapSize = 8589934592
I use the default Java settings.

Dump:
liquibase-2024_05_23T23_33_47_987.zip

Even with this one changeset, it consumes 1.5GB of RAM which is several times bigger than expected. In my case, memory consumption is the key which leads to slow execution due to memory pressure. I'd focus on RAM consumption rather than execution time.

Best regards,
Nikita