Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Added additional unicode letter points #2329

Merged
merged 5 commits into from Feb 14, 2022
Merged

Added additional unicode letter points #2329

merged 5 commits into from Feb 14, 2022

Conversation

nvoxland
Copy link
Contributor

@nvoxland nvoxland commented Jan 5, 2022

Description

Comments in #1107 listed additional unicode points which cause problems with Japanese chars in sql


Dev Handoff Notes (Internal Use)

Links

Testing

  • Steps to Reproduce: Create sql with unicode chars in the new ragnes
  • Guidance:
    • Impact: Expands what chars don't cause errors when parsed with the SQL parser

Dev Verification

Ensured existing parser-based tests pass. Did not add additional automated tests since we don't check for every possible char

┆Issue is synchronized with this Jira Bug by Unito

@nvoxland nvoxland added this to To Do in Conditioning++ via automation Jan 5, 2022
@nvoxland nvoxland moved this from To Do to Code Review in Conditioning++ Jan 5, 2022
@suryaaki2 suryaaki2 moved this from Code Review to Ready for Handoff (In JIRA) in Conditioning++ Jan 6, 2022
@XDelphiGrl XDelphiGrl assigned XDelphiGrl and nvoxland and unassigned XDelphiGrl Feb 7, 2022
@XDelphiGrl XDelphiGrl assigned XDelphiGrl and unassigned nvoxland Feb 9, 2022
@@ -31,7 +31,6 @@ class SimpleSqlGrammarTest extends Specification {
"create table /* comment here */ test" | ["create", " ", "table", " ", "/* comment here */", " ", "test"]
"insert 'a string'" | ["insert", " ", "'a string'"]
"invalid ' sql" | ["invalid", " ", "'", " ", "sql"]


"utf8-〠@chars works" | ["utf8", "-", "〠@chars", " ", "works"]
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for adding this test.

@XDelphiGrl
Copy link
Contributor

The parse error reproduces during update-sql or update with insert statements of strings containing unicode characters into an ntext or nvarchar column. During initial testing, I used inserts with a value containing one problematic unicode character; this did not show the error because there was nothing to 'parse' with a single character.

Special Thanks! Borja Serrano, your comment with a full SQL insert statement in CORE-3269 gave me what I needed to reproduce this bug. Thank you!


Example XML Changelog with Inserts

https://app.zenhub.com/files/2019791/e41ea4e8-c4df-430c-9884-0ddef2af0b41/download

  • Validate update-sql does not throw a parse error when passed unicode characters. PASS

    • degree (u00b) PASS
    • British pound (u0083) PASS
    • Japanese industrial symbol (u3004) PASS
  • Validate updatel does not throw a parse error when passed unicode characters. PASS

    • degree (u00b) PASS
    • British pound (u0083) PASS
    • Japanese industrial symbol (u3004) PASS

Test Environment
Liquibase Core: sqlgrammar-unicode/1381/3dd0fa, Pro: master/546/12ffc3
SQL Server 2017
Passing Functional Tests

@nvoxland nvoxland merged commit e9e17dc into master Feb 14, 2022
Conditioning++ automation moved this from Ready for Handoff (In JIRA) to Done Feb 14, 2022
@nvoxland nvoxland deleted the sqlgrammar-unicode branch February 14, 2022 14:48
@nvoxland nvoxland added this to the v4.8.0 milestone Feb 22, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
No open projects
Development

Successfully merging this pull request may close these issues.

None yet

3 participants