Adding lexer for CDDL

[Anteru]

(Original issue 1535 created by tdubouch on 2019-08-17T06:56:37.002511+00:00)

Concise Data Definition Language (CDDL) describes JSON or CBOR data structure and has been standardized as of RFC 8610.

A first attempt at writing a lexer is available here. The lexer is loosely based on the grammar in the RFC but is already handling most of the cases. The test file contains all of the examples from the RFC.

[birkenfeld]

Please file a pull request 😄

[hellp]

I've also worked on a CDDL lexer for pygments: https://github.com/hellp/pygments-cddl-lexer/ — not sure if I'll come back to it, but IIRC it's pretty complete, based on the RFC from March 2019.

[serianox]

Since @hellp lexer seemed much more cleaner and complete, I integrated it instead of mine (see #1379).

I had to fix one thing though, and I still don't know what happened. The first rule r'#' would send the lexer into infinite loop, and changing it to r[#] fixed the issue.

@hellp Is this ok with you?

[serianox]

Ok, just found out.
I had to set the whole regex to re.VERBOSE, because inline modifiers are now forbidden in Python re module. But then, the regex r'#' would become r'', which is indeed an infinite loop. Escaping the pound solves this... :|

[hellp]

I'm fine with whatever works. Was the behavior of inline modifiers changed recently? It worked for me, I guess.

[serianox]

@hellp I discovered they have been deprecated in Python 3.x (maybe 3.4+), and are now forbidden at least in Python 3.7.5. Hence the changes.