Release H2 2.0.202

[dyrone]

There has been a long time to release the new version of H2.

Hope to consider it~

Thanks.

[ctabin]

I agree. Can't wait for a new version to be officially released 🙏

It would be great to see H2 released more often too 😅

[dyrone]

Yeah.

I wish to fix a bug of Gerrit with the new release of H2.

So, plz consider it.

Thanks!

[yuedaxia76]

I think so too.

[grandinj]

For the next version of H2, I think we should switch to some kind of semantic type versioning, and bump the major version number to 2.

Semantic versioning is not perfect, but every little bit helps.

And this should make clear to people that they really need to do an export/import cycle before using this version.

@katzyn @andreitokar @thomasmueller ?

[thomasmueller]

Hi, Yes, bumping the major version makes sense, if there is an incompatibility. Problematic can also be "downgrade incompatibility", that is, if you open a database file from a newer version, close it, and then you open it with an older version. If this is not guaranteed to work (verified with tests), then we should prevent opening the newer database file with an older version (with a nice error message... but of course there is no way to change the error message of older versions...). I'm really sorry, I planned to work on the release, but didn't finish it so far... Regards, Thomas

…

On Wed, Jul 22, 2020 at 9:16 AM Noel Grandin ***@***.***> wrote: For the next version of H2, I think we should switch to some kind of semantic type versioning, and bump the major version number to 2. Semantic versioning is not perfect, but every little bit helps. And this should make clear to people that they really need to do an export/import cycle before using this version. @katzyn <https://github.com/katzyn> @andreitokar <https://github.com/andreitokar> @thomasmueller <https://github.com/thomasmueller> ? — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#2491 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAHDUGPKTXIWM2YN5KEFKQDR42G45ANCNFSM4LDXJYOA> .

[katzyn]

First of all, we're not yet ready to make a release from my point of view.

I think I need to finish changes in attributes of columns first, it particular, #2771 should be considered as release blocker, because current H2 clears this flag from identity columns, but we want to restore it and handle it properly. This issue is relatively easy and I hope it will be fixed soon.

Unfortunately, we still have a long-standing regression #1808. It prevents upgrades from 1.4.197 to newer versions for projects with LOBs. We need to decide is it a release blocker or not.

Due to large amount of incompatible changes we need to write a migration guide and add a link to it to all release announcements.

MVStore backend is only one-way compatible, older versions can corrupt newer databases. Sometimes newer versions don't work with older databases too. Maybe we should write build number to the header of MVStore itself, because currently CREATE_BUILD from H2 is not available early. When we read it, database can be already changed by storage backend.

Release numbers should be more meaningful, I agree. Huge releases should have changes in the major or at least in minor version. Small releases (such as 1.4.199 was) can have only different build numbers.

I also think that we need to allocate some build numbers for development releases. For example, we can use 202, 204, 206, … for official releases, and 201, 203, 205, … for snapshot builds. Current H2 still uses build number 200, but it is very different from real H2 1.4.200. We can increment build number before each release and increment it again immediately after release.

[grandinj]

I don't think #1808 is really any worse than previous revisions, it possibly just shows up more easily. LOBs are a long-standing problem child.
I have a proposed solution for that which requires a serialization format change, but no time to finish it.

[andreitokar]

Correct me, if I am wrong, but for "downgrade incompatibility" there is MVStore.FORMAT_WRITE. If we bump it up to 2, it'll prevent older versions from opening the file, and I think, we should with this release.

[katzyn]

Yes, we have FORMAT_WRITE and FORMAT_READ, with backward-incompatible changes in MVStore both fields need to be incremented, but if previous version can safely open database in read-only mode, only FORMAT_WRITE needs to be incemented.

[katzyn]

If we want to increment these fields, we also need to upgrade old database files opened in read-write mode. Because otherwise H2 can write something not compatible with older versions.

From my point of view, the best choice here is to export such database file to SQL (in many cases H2 can do that with current version even when H2 can't really work with it in normal way), create a backup copy of it, and replace it with the new database in the new format and populate it from the script.

[andreitokar]

If we want to increment these fields, we also need to upgrade old database

If by "upgrade", you mean to re-write file header with the new FORMAT_WRITE value, this suppose to happen automatically with the current code, when it erases "HDR_CLEAN" flag on first commit after start.

[katzyn]

That's good, but I'm worried a little bit about such upgrade without database recreation. In my experience, 199 can't reliably work with databases from 197, 200 can't reliably work with databases from 199. I'm talking about real production databases. After export and import everything works properly, however.

[andreitokar]

You are preaching to the choir. Of course, SQL export/import is still preferable (the only "supported") way to upgrade. It'll make everyone's life easier, including people who has to manage the upgrade. Does it mean that we should enforce it, and drop any attempts to be backward compatible? I don't think so.

[katzyn]

Current H2 can't open some databases from 1.4.200 and older versions in read-only mode, because it tries to re-build indexes on UUID columns during initialization. It doesn't look like a significant issue, however.

---

@grandinj

I also think that we need to allocate some build numbers for development releases. For example, we can use 202, 204, 206, … for official releases, and 201, 203, 205, … for snapshot builds. Current H2 still uses build number 200, but it is very different from real H2 1.4.200. We can increment build number before each release and increment it again immediately after release.

What do you think?

[grandinj]

Sounds reasonable

[katzyn]

BTW, what to do with our roadmap? It mentions planned changes for 1.5, planned changes for 1.4. Some items are obscure. Maybe remove this page? With issue tracker it became obsolete.

[katzyn]

H2 1.4.200 silently opens databases from current version as empty ones. So we definitely need to increment not only the MVStore.FORMAT_WRITE, but also MVStore.FORMAT_READ.

[grandinj]

Yeah, dump the roadmap, no longer useful

[katzyn]

#2812 is a release blocker. This issue was was always there, but after fixes of other issues it became much more visible, because equality and IN conditions now also affected, previously only range comparisons were insane. It can be fixed in relatively easy way, but if we want to avoid performance loss we need to allow range comparisons on data types with the same ordering rules. And here our compatibility settings (not related with compatibility modes) create a mess for all binary string data types and data types based on a byte array (JAVA_OBJECT, JSON, GEOMETRY, and also for long+long-based UUID). Byte arrays in H2 can be compared in different data types in different ways depending on values of these settings. Because BINARY_COLLATION was scheduled for removal in the next series of releases in roadmap in 2014 and now we want to do a major release I think it's about a time to drop it (and UUID_COLLATION too, it doesn't look like something really useful).

[grandinj]

Yeah, BINARY_COLLATION and UUID_COLLATION can go if it helps resolve that

[yuedaxia76]

@ thomasmueller , hi thomasmueller Do you have a release plan?There has been a long time no release h2.

[katzyn]

The upcoming H2 2.0.202 is not compatible with 1.4.200 and cannot be released that easy.

Unfortunately, we still have a couple of significant regressions to be fixed and migration guide isn't yet written.

[dyrone]

The upcoming H2 2.0.202 is not compatible with 1.4.200 and cannot be released that easy.

Unfortunately, we still have a couple of significant regressions to be fixed and migration guide isn't yet written.

Feel the complexity of your reply and thanks for the hard work.

Wish problems could be fixed smoothly.

[katzyn]

Regression with comparison operations is now fixed. Not yet merged PR #2945 contains fixes for RUNSCRIPT command to allow data import from 1.4.200.

1. We still need a migration guide due to outstanding number of changes between 1.4.200 and 2.0.
2. We need to decide something about automatic upgrade from 1.4 binary databases. H2 performs some fixes on the fly, but even with them we can't be sure that patched database file is consistent. We can be sure only with a new database file populated with the old data.

[grandinj]

The safest thing to do, is to simply disallow automatic upgrade, and refuse to open such databases in 2.0

[katzyn]

I think H2 2.0 should be able to read 1.4 databases, but probably should not try to write anything to them.

For better user experience H2 can try to copy all data into new database file and replace old file with a new one with backup copy of the old file if data was copied successfully. Such implementation should be relatively simple and with it we can remove current tricks in indexes and other places where H2 tries to fix existing problems.

[xresch]

From my perspective as a programmer using H2, I think one interesting way could be to have some command line tool and a small programming interface for the upgrades.

For example on the command line it could look like this:

//Automatic
java -cp h2*.jar org.h2.tools.Migrate -dbfile="pathto/dbfile.mv.db" -auto

//Customized
java -cp h2*.jar org.h2.tools.Migrate -dbfile="pathto/dbfile.mv.db" -newFile="path/todbfile.mv.db" -migrateType="tablename, columnname, BINARY, VARBINARY" -migrateType="tablename, columnname, ARRAY, VARCHAR ARRAY" 

In code the programming interface could look like this:

// Automatic
// causes automatic migration of the database with default type assumptions
H2Migration.toggleAutoMigration(true);

// Customized
// Migration triggered automatically, default types used for columns not specified 
H2Migration.toggleAutoMigration(true);
H2Migration.setMigrationConfig(
     new H2MigrationConfig(oldVersion, targetVersion)
              .migrateType(tablename, columnname, "BINARY", "VARBINARY")
              .migrateType(tablename2, columnname2, "BINARY", "BINARY(12)")
              .migrateType(tablename3, columnname3, "ARRAY", "VARCHAR ARRAY")
):

//Auto Upgrade Off
// Would cause an error if older version is detected.
// Useful if the programmer wants to provide it's own migration script based on the shell interface, for example if the application has a huge database
H2Migration.toggleAutoMigration(false);
H2Migration.errorHandler(new ErrorHandler(){
     function void handle(Exception e){
          //log message etc...
     }
});

The above configs would be done before the H2Server is started or connection is established.
this is just a suggestion, it's totally up to you how you want to do it.
Looking forward to a great next release, thanks for all your efforts. :-)

[dyrone]

@katzyn Well Done!

[gruenich]

What's your current plan concerning a new release?

[katzyn]

#3016 is release blocker and it needs the new implementation of #3037 to avoid further regressions.

The upgrade documentation is also not yet ready. H2 2.0 is not compatible with 1.4 by many aspects and people need to know what to expect.

[ryanhamilton]

Worrying Signs:

• No release in a year
• Significant addition of new functionality being suggested as a blocker
• One major contributor doing 90%+ of the work

In the past I've seen this kind of scenario destroy an open source project as at some point that one contributor leaves, 1-2 persons appear wanting to pick it up but:

• They don't want to try and understand all the code since the last release
• They mostly want their bugs fixed
• Since there wasn't a release, they feel free to dump the inbetween code not in use generally
• This causes a fork and the entire project stops as there are now multiple versions.

I would advise not adding any more functionality and immediately getting wider releases happening.
In fact I'll put my money where my mouth is and offer $100 for a maven release.

(I would also take this opportunity to say thank you for your hard work @katzyn)

[xresch]

@katzyn Ryan has listed some interesting points.
What would you need to create the next release lets say in the next 2 months?
Could crowdfunding help you or getting additional developers on the project?

Also thanks from my side for all the efforts on creating a gorgeous h2 database!

[katzyn]

No release in a year

H2 doesn't have patch releases with some exclusions (such as 1.4.199). For major releases large intervals are rather normal.

Significant addition of new functionality being suggested as a blocker

This statement is wrong, only regressions in the maximum size of GEOMETRY (and JSON) and the lack of migration guide are release blockers. But they can't be properly resolved with trivial changes, they require some work, including work on problems with LOBs, these problems became more visible in the few last releases.

What would you need to create the next release lets say in the next 2 months?

A lot of free time, but I don't have it. I'm preparing fixes for the mentioned regressions and migration guide, but very slowly.

Could crowdfunding help you or getting additional developers on the project?

H2 is developed by the community and doesn't have any full-time developers. Hypothetically you can hire someone to do some specific job, but project's maintainers and other active members are unlikely available to hire and other developers aren't familiar with H2.

[xresch]

@katzyn thanks for the clarifications.

Would it be an option to setup a crowdfunding to get you more free time to work on H2? (not sure how your situation is)
We already have @ryanhamilton who would put some money on the table and I also would consider it.

[katzyn]

No, it wouldn't.

[xresch]

okay, no problem.

Would it be an option to create some "beta" release of the current code?
Other projects like jetty do this as well:
https://mvnrepository.com/artifact/org.eclipse.jetty/jetty-server

You could make newest feature available to others through maven, while you aren't responsible for any stuff that goes wrong as you marked it as "beta".
With this developers don't have to wait for the newest code until all the work is done.
You could even make a 3 or 6 month schedule and give people a a date when the next beta release will be published.

I currently wouldn't mind trying a beta version with issues on GEOMETRY/JSON and no migration guide, I don't use these two types and I can start with a new database file.

[gruenich]

What about sponsoring the remaining blockers, as described in #3048? Could we pay a company / consultant to do the heavy lifting and you guys have only be part of the integration?

[katzyn]

@xresch
There is a big difference between a web server and database system. Database files, created by snapshot versions of H2, may be silently corrupted beyond any possibility to recover their content by the next snapshot build, for example.

Actually H2 had snapshot builds a long time ago, but H2 also used different release strategy with tiny releases.

@gruenich
Every major contribution need to be discussed with the community first and should be written in such way that will not create problems in the future.

I'll fix the regression with large geometries in the next few weeks, I already wrote about a half of necessary code, but I have no idea who may help with the LOB storage of H2 that will be used for this data type too. It has many long-standing issues, situation was definitely improved since the last release, but I'm afraid it still may throw unexpected exceptions in some cases, because some still used reference to a LOB was expired. Maybe these issues aren't very critical, I don't know. It needs to be tested.

[xresch]

@katzyn ah I see, issues might not be obvious but permanent in the database file, that really would be an issue.
I understand now why beta releases are not a great idea for a database.

Keep up the good work, even if it takes a while I'm looking forward to a great next release. :-)

[Typografikon]

Any news about 2.0.202 release? I really awaiting new relase because only way to workaround #2922 issue is to use old 1.4.197 version. Please provide some info about current state of work. Thanks!

[mstroeve]

Same question here about a new release.
We are having serveral chunck not found issues with the .200 version.

[yuedaxia76]

Is there a definite release date?

[srakonjac-nq]

Hello, I think I'm not the only one that's interested in a new release (see #3202). Are there any plans to make a realise containing the fix for #3199 (an issue marked as Severity 9 Threat by Nexus Sonatype)?

Thank you and cheers!

[Sascha82]

Can we get any official feedback on this topic? If there is not quick fix in place, then we have to start with the anlysis, how this component can be replaced with a more secured one...

Thanks and best regards,
Sascha

[andreitokar]

Not sure what do you mean by "official", since this project is run purely by volunteers, but if you are looking for a really quick fix, feel free to build your own 2.0.201-SNAPSHOT jar from master branch, and post it to your local repository, if necessary.
Fix for the issue #3199 is already on master branch, and we are working on a new release.

[andreitokar]

2.0.202 release is coming! While "official" version will likely appear on Maven early next week, final artefacts are already available from Github release page and via h2database.com

[xresch]

Nice, thanks to the whole team for all the efforts you had creating this release!
Looking forward to upgrade to the newest version. :-)

[SubJunk]

Exciting news! Well done on getting there guys