Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

mysql unicode #1453

Closed
Freed-Wu opened this issue May 18, 2020 · 3 comments
Closed

mysql unicode #1453

Freed-Wu opened this issue May 18, 2020 · 3 comments

Comments

@Freed-Wu
Copy link

mysql allow user to use unicode character to create table, but in minted

\documentclass{ctexbook}
\usepackage{minted}
\begin{document}
\begin{minted}{mysql}
create table basic(
	股票编号 int not null,
	上市年限 int not null,
	所属行业 varchar(20) not null,
	所属概念板块 varchar(200) not null
);
\end{minted}
\end{document}

the result:

image

the unicode character be considered as PYG@token@err which will be added a red box. is it true?

@marijnschraagen
Copy link

Interestingly, it does work if there is a non-space latin character before the unicode characters:

\documentclass{ctexbook}
\usepackage{minted}
\begin{document}
\begin{minted}{mysql}
create table basic(
   a股票编号 int not null,
   b上市年限 int not null,
   c所属行业 varchar(20) not null,
   d所属概念板块 varchar(200) not null
);
\end{minted}
\end{document}

Schermafdruk van 2020-06-23 15-58-29

@Freed-Wu
Copy link
Author

amazing!

@kurtmckee
Copy link
Contributor

I'm working on an overhaul of the MySQL lexer. I will try to make this work, at least to the point that it doesn't render as an error.

kurtmckee added a commit to kurtmckee/pr-pygments that referenced this issue Aug 29, 2020
kurtmckee added a commit to kurtmckee/pr-pygments that referenced this issue Aug 29, 2020
kurtmckee added a commit to kurtmckee/pr-pygments that referenced this issue Aug 29, 2020
Fixes pygments#975, pygments#1063, pygments#1453

Changes include:

Documentation
-------------

* Note in the lexer docstring that Oracle MySQL is the target syntax.
  MariaDB syntax is not a target (though there is significant overlap).

Unit tests
----------

* Add 140 unit tests for MySQL.

Literals
--------

* Hexadecimal/binary/date/time/timestamp literals are supported.
* Integer mantissas are supported for scientific notation.
* In-string escapes are now tokenized properly.
* Support the "unknown" constant.

Comments
--------

* Optimizer hints are now supported, and keywords are
  recognized and tokenized as preprocessor instructions.
* Remove nested multi-line comment support, which is no
  longer supported in MySQL.

Variables
---------

* Support the '@' prefix for variable names.
* Lift restrictions on characters in unquoted variable names.
  (MySQL does not impose a restriction on lead characters.)
* Support single/double/backtick-quoted variable names, including escapes.
* Support the '@@' prefix for system variable names.
* Support '?' as a variable so people can demonstrate prepared statements.

Keywords
--------

* Keyword / data type / function are now in a separate, auto-updating file.
* Support 25 additional data types (including spatial and JSON types).
* Support 460 additional MySQL keywords.
* Support 372 MySQL functions.
  Explicit function support resolves a bug that causes non-function
  items to be treated as functions simply because they have a trailing
  opening parenthesis.
* Support exceptions for the 'SET' keyword, which is both a datatype and
  a keyword depending on context.

Schema object names
-------------------

* Support Unicode in MySQL schema object names.
* Support parsing of backtick-quoted schema object name escapes.
  (Escapes do not produce a distinct token type at this time.)

Operators
---------

* Remove non-operator characters from the list of operators.
* Remove non-punctuation characters from the list of punctuation.
kurtmckee added a commit to kurtmckee/pr-pygments that referenced this issue Aug 29, 2020
Fixes pygments#975, pygments#1063, pygments#1453

Changes include:

Documentation
-------------

* Note in the lexer docstring that Oracle MySQL is the target syntax.
  MariaDB syntax is not a target (though there is significant overlap).

Unit tests
----------

* Add 140 unit tests for MySQL.

Literals
--------

* Hexadecimal/binary/date/time/timestamp literals are supported.
* Integer mantissas are supported for scientific notation.
* In-string escapes are now tokenized properly.
* Support the "unknown" constant.

Comments
--------

* Optimizer hints are now supported, and keywords are
  recognized and tokenized as preprocessor instructions.
* Remove nested multi-line comment support, which is no
  longer supported in MySQL.

Variables
---------

* Support the '@' prefix for variable names.
* Lift restrictions on characters in unquoted variable names.
  (MySQL does not impose a restriction on lead characters.)
* Support single/double/backtick-quoted variable names, including escapes.
* Support the '@@' prefix for system variable names.
* Support '?' as a variable so people can demonstrate prepared statements.

Keywords
--------

* Keyword / data type / function are now in a separate, auto-updating file.
* Support 25 additional data types (including spatial and JSON types).
* Support 460 additional MySQL keywords.
* Support 372 MySQL functions.
  Explicit function support resolves a bug that causes non-function
  items to be treated as functions simply because they have a trailing
  opening parenthesis.
* Support exceptions for the 'SET' keyword, which is both a datatype and
  a keyword depending on context.

Schema object names
-------------------

* Support Unicode in MySQL schema object names.
* Support parsing of backtick-quoted schema object name escapes.
  (Escapes do not produce a distinct token type at this time.)

Operators
---------

* Remove non-operator characters from the list of operators.
* Remove non-punctuation characters from the list of punctuation.
kurtmckee added a commit to kurtmckee/pr-pygments that referenced this issue Aug 31, 2020
Fixes pygments#975
Fixes pygments#1063
Fixes pygments#1453

Changes include:

Documentation
-------------

* Note in the lexer docstring that Oracle MySQL is the target syntax.
  MariaDB syntax is not a target (though there is significant overlap).

Unit tests
----------

* Add 140 unit tests for MySQL.

Literals
--------

* Hexadecimal/binary/date/time/timestamp literals are supported.
* Integer mantissas are supported for scientific notation.
* In-string escapes are now tokenized properly.
* Support the "unknown" constant.

Comments
--------

* Optimizer hints are now supported, and keywords are
  recognized and tokenized as preprocessor instructions.
* Remove nested multi-line comment support, which is no
  longer supported in MySQL.

Variables
---------

* Support the '@' prefix for variable names.
* Lift restrictions on characters in unquoted variable names.
  (MySQL does not impose a restriction on lead characters.)
* Support single/double/backtick-quoted variable names, including escapes.
* Support the '@@' prefix for system variable names.
* Support '?' as a variable so people can demonstrate prepared statements.

Keywords
--------

* Keyword / data type / function are now in a separate, auto-updating file.
* Support 25 additional data types (including spatial and JSON types).
* Support 460 additional MySQL keywords.
* Support 372 MySQL functions.
  Explicit function support resolves a bug that causes non-function
  items to be treated as functions simply because they have a trailing
  opening parenthesis.
* Support exceptions for the 'SET' keyword, which is both a datatype and
  a keyword depending on context.

Schema object names
-------------------

* Support Unicode in MySQL schema object names.
* Support parsing of backtick-quoted schema object name escapes.
  (Escapes do not produce a distinct token type at this time.)

Operators
---------

* Remove non-operator characters from the list of operators.
* Remove non-punctuation characters from the list of punctuation.
Anteru pushed a commit that referenced this issue Sep 6, 2020
* Overhaul the MySQL lexer

Fixes #975, #1063, #1453

Changes include:

Documentation
-------------

* Note in the lexer docstring that Oracle MySQL is the target syntax.
  MariaDB syntax is not a target (though there is significant overlap).

Unit tests
----------

* Add 140 unit tests for MySQL.

Literals
--------

* Hexadecimal/binary/date/time/timestamp literals are supported.
* Integer mantissas are supported for scientific notation.
* In-string escapes are now tokenized properly.
* Support the "unknown" constant.

Comments
--------

* Optimizer hints are now supported, and keywords are
  recognized and tokenized as preprocessor instructions.
* Remove nested multi-line comment support, which is no
  longer supported in MySQL.

Variables
---------

* Support the '@' prefix for variable names.
* Lift restrictions on characters in unquoted variable names.
  (MySQL does not impose a restriction on lead characters.)
* Support single/double/backtick-quoted variable names, including escapes.
* Support the '@@' prefix for system variable names.
* Support '?' as a variable so people can demonstrate prepared statements.

Keywords
--------

* Keyword / data type / function are now in a separate, auto-updating file.
* Support 25 additional data types (including spatial and JSON types).
* Support 460 additional MySQL keywords.
* Support 372 MySQL functions.
  Explicit function support resolves a bug that causes non-function
  items to be treated as functions simply because they have a trailing
  opening parenthesis.
* Support exceptions for the 'SET' keyword, which is both a datatype and
  a keyword depending on context.

Schema object names
-------------------

* Support Unicode in MySQL schema object names.
* Support parsing of backtick-quoted schema object name escapes.
  (Escapes do not produce a distinct token type at this time.)

Operators
---------

* Remove non-operator characters from the list of operators.
* Remove non-punctuation characters from the list of punctuation.

* Cleanup items based on feedback

* Remove an unnecessary optional newline lookahead for single-line comments
@Anteru Anteru closed this as completed Sep 6, 2020
Kenny2github pushed a commit to Kenny2github/pygments that referenced this issue Sep 22, 2020
* Overhaul the MySQL lexer

Fixes pygments#975, pygments#1063, pygments#1453

Changes include:

Documentation
-------------

* Note in the lexer docstring that Oracle MySQL is the target syntax.
  MariaDB syntax is not a target (though there is significant overlap).

Unit tests
----------

* Add 140 unit tests for MySQL.

Literals
--------

* Hexadecimal/binary/date/time/timestamp literals are supported.
* Integer mantissas are supported for scientific notation.
* In-string escapes are now tokenized properly.
* Support the "unknown" constant.

Comments
--------

* Optimizer hints are now supported, and keywords are
  recognized and tokenized as preprocessor instructions.
* Remove nested multi-line comment support, which is no
  longer supported in MySQL.

Variables
---------

* Support the '@' prefix for variable names.
* Lift restrictions on characters in unquoted variable names.
  (MySQL does not impose a restriction on lead characters.)
* Support single/double/backtick-quoted variable names, including escapes.
* Support the '@@' prefix for system variable names.
* Support '?' as a variable so people can demonstrate prepared statements.

Keywords
--------

* Keyword / data type / function are now in a separate, auto-updating file.
* Support 25 additional data types (including spatial and JSON types).
* Support 460 additional MySQL keywords.
* Support 372 MySQL functions.
  Explicit function support resolves a bug that causes non-function
  items to be treated as functions simply because they have a trailing
  opening parenthesis.
* Support exceptions for the 'SET' keyword, which is both a datatype and
  a keyword depending on context.

Schema object names
-------------------

* Support Unicode in MySQL schema object names.
* Support parsing of backtick-quoted schema object name escapes.
  (Escapes do not produce a distinct token type at this time.)

Operators
---------

* Remove non-operator characters from the list of operators.
* Remove non-punctuation characters from the list of punctuation.

* Cleanup items based on feedback

* Remove an unnecessary optional newline lookahead for single-line comments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

4 participants