fix #1106 #CORE-3269 degree sign accepted as text

[matteoturra]

---

name: Pull Request
about: Create a report to help us improve
title: ''
labels: Status:Discovery
assignees: ''

---

Environment

Liquibase Version: 3.8.6

Database Vendor & Version: Oracle 12

Operating System Type & Version: Windows 10

Pull Request Type

• [ x] Bug fix (non-breaking change which fixes an issue.)
• Enhancement/New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)

Description

The sqlformat parser does not accepts certain character from Latin1 Iso extension (like £°) throwing TokenMgrError
See: https://liquibase.jira.com/browse/CORE-3269

Steps To Reproduce

Create a changeset with an insert statement like
insert into table (column) values ('warm temp. 32°')

See #1106

Fast Track PR Acceptance Checklist:

• [ x] Build is successful and all new and existing tests pass
• [ x] Added Unit Test(s)
• Added Integration Test(s)
• Documentation Updated

[ro-rah]

Hi @matteoturra ,

Many thanks for your PR submission!

Would you mind filling out the description template for this PR? We use this to assess the PRs before they enter into internal Engineering. It looks like these items like description of the issue being fixed, why you updated the files, version of Liquibase, and steps to reproduce would be helpful to. This is what step 9 details in our community guide to submitting PRs.

Thanks for creating unit test!

Again, thanks for all the work you have put in so far.

-Ronak

[ro-rah]

Hi @matteoturra ,

Thanks for writing the Unit tests, and I see this should close #1106 as well as closes CORE-3269. The unit test speaks for itself and the test can be the following insert should not cause a parse error:
insert into table (column) values ('warm temp. 32°')

I'll get this into conditioning.

[ro-rah]

Reasoned on the Risk low b/c it contains unit tests, and it is currently broken.

[Aeolun]

@ro-rah It's been months and it appears this is still an issue. I'm currently unable to deal with fullwidth tilde ～. Is there any chance to get this merged?

[Aeolun]

There are actually a lot more ranges that break raw SQL when you are working with the Japanese language.

[ro-rah]

Hi @Aeolun and @matteoturra, escalating to my colleague @molivasdat for update.

[nvoxland]

This change was already made at some point, but I created #2329 for the additional unicode points @Aeolun mentioned

[kataggart]

Thank you @nvoxland !