how to force a token type for mis-lexed parts

[goyalyashpal]

hi!

title/summary

how to force a token type for mis-parsed parts

details

• i am using mysql cli session with psql (postgresql shell) lexer as there's no lexer for mysql cli session.
• often the lexing performs badly for one reason or the other.
• so, in cases where it can't be fixed with hacks, i'd like to be able to force a token scope (like prompt, output, etc)/token type (like Generic.Output, etc) on whole of the given text...
  similar to how there's guess mode for lexers, but then there's also ability to specify/force a lexer

orrrr..... maybe if there can be some other clever hack like say modifying the input file in some way to directly instruct pygments in some way.

related issues:

i remember reading somewhere something like "look into correctly parsing statement end comments" but i can't recall where.

pygments/pygments/lexers/sql.py

Lines 342 to 347 in 0f3ddb3

	# Check if this is the end of the command
	# TODO: better handle multiline comments at the end with
	# a lexer with an external state?
	if re_psql_command.match(curcode) \
	or re_end_command.search(curcode):
	break

regex:

pygments/pygments/lexers/sql.py

Lines 271 to 278 in 0f3ddb3

	re_prompt = re.compile(r'^(\S.*?)??[=\-\(\$\'\"][#>]')
	re_psql_command = re.compile(r'\s*\\')
	re_end_command = re.compile(r';\s*(--.*?)?$')
	re_psql_command = re.compile(r'(\s*)(\\.+?)(\s+)$')
	re_error = re.compile(r'(ERROR|FATAL):')
	re_message = re.compile(
	r'((?:DEBUG|INFO|NOTICE|WARNING|ERROR|'
	r'FATAL|HINT|DETAIL|CONTEXT|LINE [0-9]+):)(.*?\n)')

regex summary relevant here:

• prompt's delimiter: ->
• end_command delimiter: ;

versions:

• pygments: now: 2.17.2 (at the time of filing: 2.16.1)
• python: 3.11.2

---

samples

following are the samples given in reverse chronological order:

meta delimiters (can't fix)

changes statement end delimiter from ; to \G (vertical output) or \p (print command), regex keeps on looking for ;, bleeds outside till next query.

the \p is very useful when using with source (\s).

also, again as with \help it capitalizes these above too, which is very bad in this context, as it contains normal queries, where keywordcase=upper filter would be very much required.

mysql-> select 2 + 2 \G
mysql-> select 2 + 2 \p\G
mysql-> select 2 + 2 \p\G; -- renders correct in pyg, causes error in mysql

procedures (can't fix)

the procedure body uses delimiter ;, regex stops there; marks the remaining portion of query too as Generic.Output

mysql-> delimiter ;;
mysql-> create procedure if not exists
    -> rectarea (in length int, in breadth int, out area int)
    -> begin
    ->     set area = length * breadth;
    ->     select area;
    -> end;;
Query OK, 0 rows affected (0.01 sec)

mysql-> delimiter ;

separately treat help queries with keywordcase=lower filter

the "top" one produces wrong output, following one does correct

mysql-> \help contents; -- filter keywordcase=upper as \HELP doesn't work here
mysql-> \help contents; -- filter keywordcase=lower

chang prompt to match regex

mysql> select * from db.mtable;
mysql-> select * from db.mtable;

screenshot (for all the above samples:

[goyalyashpal]

for the time being, i reckon that i can also use "forced frozen cache" after semi-manually getting the tokenized result to be just like i want it to be. but that has to be done at the very last, as the document i am working on at the moment is going fast large changes

[goyalyashpal]

i think the two of these (with procedure, & delimiters other than ;) both can be fixed by lexing theee input line by line. so, everything with -> in start to be lexed as command input , rest becomeees output...

it will only eeerror out in multilin comments, ... which itself i thiink can be easily avoided.