GitHub - lib-ruby-parser/mri-tokenizer

MRI tokenizer

This repo contains a set of scripts and MRI patches to build a pure zero-dependency tokenizer that is based on MRI's grammar.

Compiled binary is attached to release called latest. Only x86-64 Linux and macOS are supported.

Name		Name	Last commit message	Last commit date
Latest commit History 6 Commits
.github		.github
test		test
README.md		README.md
mri-tokenizer.c		mri-tokenizer.c
mri-tokenizer.mk		mri-tokenizer.mk
patch-parse-y.rb		patch-parse-y.rb