New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Lua scanner -- Meta-commit. #22
Conversation
This commit is a super-commit containing all the subcommits for implementing the Lua scanner.
Any opinion? Plans for merging, rejecting? Valete, |
If noone comments, I cannot improve the code. And as far as I know there’s always something to improve ;-) Valete, |
Quintus: Sorry for taking so long to answer. I will look at it, Lua is on top of my wish list :) |
Wow, your scanner seems almost perfect! It had no problems with the example file I threw at it (http://svn.rubychan.de/coderay-scanner-tests/trunk/lua/example.in.lua). I fixed a Ruby 1.8 incompatibility (leading method dots…I was against that change in ruby-core ;-), and escaped the Please run rake test:scanner:lua on the I wonder, why did you choose to use |
You see I’m no better ;-). We all are forced to spent our time elsewhere...
Thanks ;-). Given my initial motivation of writing this scanner (I justed wanted to highlight Lua code in a scripting manual generated via kramdown and was surprised that CodeRay had no Lua support) this surprises me as well. Of course I’ve tested some obscure code with the scanner (did you know all the ways you can write numbers in Lua?), but never with such a big file. Thanks for testing!
OK... I’ve stopped caring about 1.8 compatibility as this version will soon officially be discouraged.
So your editor is not CodeRay-powered! What a shame! :-D
I’ve then created the file with this content:
Not sure what you mean, looks quite identical to me. Could you be more specific?
Mh, I confess to not have thought that CodeRay not only aims to be the best, but also the fastest syntax highlighter out there in Ruby land... I simlpy dislike super-long case statements as they decrease readability which I favour over performance. Plus, it is easier to make recursive calls with methods rather than case statements. I can try to rewrite the scanner using Vale, |
I am prepared to drop 1.8 compat after CodeRay 1.1, but for the time being, I would like to keep it.
Technically, they have to be escaped. It seems TextMate is trying to be too clever when it parses RegExps.
Fixed. Forgot how to use SVN.
It's identical because it creates So, in your case, look at either of the files and see if you can spot any problems.
I'm sure we could talk over this forever, exchanging valid points. Can you accept that I want to keep the codebase single-style? As in, all scanners should look/work the same, anticipating another batch rewrite in the future. |
Not if you store the state in a variable anyway.
I already did that in |
One thing I'm wondering about is the addition of the "table" Style. Afaik, the tables in Lua are somewhat similar to the Arrays in PHP, supporting both arrays and hashes/associative arrays. Since styles are across languages in Coderay, I was wondering if there is a more appropriate name for the style itself? (or another style that could be reused) |
I like the shades-of-blue trick. We could use that for JSON, too. Adding a new token kind is fine; maybe we find a more generic name? |
Yah, the shades-of-blue trick is cool. It reminds me of how Xcode highlights indentation with shades of gray. Maybe it could be named a It's been a while since I've looked at Lua, but I'm not seeing any obvious issues with the tokenizer in
I've merged master and faster into lua-scanner, along with escaping some additional regex expressions. I'm working to convert a Lua blog from WordPress to NestaCMS, so I'll be referencing this branch. Nathan. |
Changed :table to :map, which is better for reuse in other languages, and won't be confused with :table based html rendering. 279348d
|
This article has some slightly bizarre code that is reporting as errors with the Lua scanner: http://www.luanova.org/porting-lua-to-risc-os. |
Good finding! Can you add that code to the Lua test suite? |
I'm not actually sure if that is valid Lua code, I just assume it is because someone published it on my web site :-) |
> luac -
-- Fred, tiny wimp program using error dialogue
require "wimp.task"
do
local dim,! in riscos
luac: stdin:6: '<name>' expected near '!' |
FWIW I tried manually patching the current gem with the code from this commit, and the process using highlighting hung when I tried to highlight a Lua snippet. In case it was snippet-specific, here's what I was coloring:
|
Thanks! Can you post the error message? |
@korny I wish I could provide an error message, but there was none. Literally the process took 100% CPU and stayed there until I killed it. :/ |
FWIW, I've written my own simple Lua scanner here: https://github.com/Phrogz/coderay/blob/master/lib/coderay/scanners/lua.rb (I started out writing a recursive descent parser for Lua before realizing that the EBNF grammar supplied is left-recursive in many locations. So I went with the one huge regex approach that covers 90% of the formatting I wanted.) |
Done! Wow, activity after a few months ;) Can we still get this ready for CodeRay 1.1? I'm trying to… |
@Phrogz: I added your snipped to the test suite, it runs smoothly. Can't produce a bug. |
Yes, it was Latin-1 or something. Fixed it.
Funny, this is fixed by moving the
Lua says no:
Still, I added it as an example. |
Yay, I have no issue left. The scanner is a bit slower than the Ruby scanner, but it's complex and handles complicated Lua code very well. We can optimize it later. Thank you, @Quintus, for your awesome work on this. It's sad that my laziness prevented it from being merged earlier. CodeRay 1.1 will have great Lua support thanks to you! |
Thanks @korny |
OK, here’s the "meta-commit" for the Lua scanner you requested.
But I don’t think it’s unusual to have a pull request containing multiple commits. In fact it’s what I see most times when I look at pull requests (GitHub even encourages to add even more commits to a pull requests after it has been opened).
Vale,
Quintus