Re: Regular expressions in lexing and parsing

On Tue, Jun 18, 2019 at 12:23 PM John wrote:
> [I agree with the sentiment to use formal grammars for both lex and
> parse, but it always has seemed to me that running them separately as
> coroutines makes it easier to deal with comments and whitespace
> without having to hang them on every token definition. I realize that
> the more context sensitive the language, the more it is likely to make
> sense to combine them. -John]


The Meta-S system has a way to do that automatically. The "whitespace
rule" (__ws is the reserved name for it) also has two alias sequences
that make it simple to specify optional versus required whitespace (#@
and ## respectively):


some_rule ## some_other_rule // whitespace is not optional


some_rule #? "(" #? some_other_rule #? ")" // whitespace in all
positions is optional


Moreover, the definition of the __ws can be:


__ws ::= #notree #standard_whitespace; // defined as standard C type
whitespace, and don't stick it on the AST when recognized in the input


I have found that, while sometimes verbose, this explicit handling
makes the dynamic nature of the grammars far easier to implement at
the back-end. (At the cost of the human grammar author having to think
about whitespace more often than some would prefer.)


If I recall correctly, it was Hans-Peter Dietrich (the OP) who
convinced me a number of years back to include the ability to specify
certain tokens that did not need explicit whitespace around them in
the productions that use them. (The grammar compiler itself then takes
responsibility for inserting the appropriate whitespace operators
behind the scenes during compilation when those tokens are used.)


In the long run, this hybrid approach has offered the most
versatility, without being too cumbersome. (There are bigger dragons
to fight with adaptive systems that can modify their rules in
mid-parse.....)


--
Quinn Jackson CSci MIScT SMACM SMIEEE FRSA


LinkedIn: http://ca.linkedin.com/in/quinnjackson/
ResearchGate: http://researchgate.net/profile/Quinn_Jackson/

Return to the comp.compilers page.
Search the comp.compilers archives again.