Re: State-of-the-art algorithms for lexical analysis?

On 6/6/22 8:16 PM, Christopher F Clark wrote:


> In fact, there is only thing that I have not seen a DFA lexer do that I think is
> worth doing at the lexical level (and not via a screener). That is recognizing
> tokens the start with a length prefix, e.g. 10Habcdefhij. Such tokens are
> common in things like network protocols and they would be relatively easy
> to implement, but I've not seen it done.


I'm not sure what you mean. The nnH syntax has to be included into
general number syntax (like 0x... or nnE...).


Or do you mean a token built from the next nn input characters? In this
case both a lower and upper bound were interesting for e.g. (recognized)
identifier length or distinction of Unicode codepoint formats.


> Beyond that it is my relatively firm belief that one should almost always
> have only simple regular expressions, e.g. that the one for floating point
> numbers should be one of the most complex ones. Otherwise you are trying
> to do too much in the scanner. And you are asking for trouble when you do.


ACK


DoDi
[I believe he means Fortran style Hollerith strings, where the number says
how many characters are in the following string. The number is just a count,
not semantically a number in the language. DFAs can't do that other than by
enumerating every possible length. -John]

Return to the comp.compilers page.
Search the comp.compilers archives again.