Re: State-of-the-art algorithms for lexical analysis?

On Sunday, June 5, 2022 at 2:08:12 PM UTC-7, Roger L Costello wrote:


(snip)


> Are regular expressions still the best way to specify tokens?


Some years ago, I used to work with a company that sold hardware
search processors to a certain three letter agency that we are not
supposed to mention, but everyone knows.


It has a completely different PSL, Pattern Specification Language,
much more powerful than the usual regular expression.


Both the standard and extended regular expression are nice, in that we
get used to using them, especially with grep, and without thinking too
much about them.


I suspect, though, that if they hadn't previously been defined, we
might come up with something different today.


Among others, PSL has the ability to define approximate matches,
such as a word with one or more misspellings, that is insertions,
deletions, or substitutions. Usual RE don't have that ability.


There are also PSL expressions for ranges of numbers.
You can often do that with very complicated RE, considering
all of the possibilities. PSL automatically processes those
possibilities. (Some can expand to complicated code.)


I suspect that in many cases the usual RE is not optimal for
lexical analysis, other than being well known.


But as noted, DFA are likely the best way to do them.


Though that could change with changes in computer hardware.

Return to the comp.compilers page.
Search the comp.compilers archives again.