Re: What does it mean to "move characters" in the lexer?

Kaz Kylheku <480-992-1380@kylheku.com> schrieb:


> I remember reading some article some years ago whereby some Javascript
> programmer discovered it was faster to read JSON from a file using
> dedicated JSON routines available in Javascript, than to declare the
> same syntax in the Javascript program as a literal and let it be
> scanned along with the program and available to it that way.


This came up on comp.arch recently.


There is an insanely fast JSON parser ad UTF-8 validator based
on SIMD to be found at https://github.com/simdjson/simdjson .
They select a different length of vector according to
the CPU version they find. The algorithm is described at
https://arxiv.org/pdf/1902.08318.pdf. It
heavily relies on special-casing for JSON and for the SIMD
instructions that are available.


A general SIMD-based parser generator is likely to be even harder
to write and will probably not outperform the package above (nor,
for that case, a traditional character-at-a-time approach).


Is there research on this?

Return to the comp.compilers page.
Search the comp.compilers archives again.