An ingenious scanning problem!

Hi Folks,


Page 115-116 of the Flex User's Manual [1] has an ingenious problem posed by
Jan Kort on 05 September 1998.


Here's a description of the problem:


The input contains this: TEST1\n


That is, the input contains the string TEST1 followed by a newline.


The Flex scanner contains a rule that matches on the input:


TEST1\n { action }


Here's a description of its action:


Of the text that was matched, I only
want the first 5 characters (i.e., TEST1).


That action is expressed in Flex this way:


TEST1\n { yyless(5); }


In the scanner there are also two newline rules.


One newline rule matches on a newline at the beginning of a line (i.e., empty
line):


^\n { action }


The other newline rule matches on a newline at the end of a line:


\n { action }


Question: After the action for TEST1\n is executed, which newline rule is
executed?


The answer depends on the meaning of yyless(). Here are two possible meanings
for yyless():


(a) The scanner is backing up in the input stream.
(b) The scanner is pushing new characters onto the input stream.


If the meaning of yyless() is (a), then the second newline rule will be
executed.


If the meaning of yyless() is (b), then the first newline rule will be
executed.


So which does Flex do?


............................


Answer: (b)


Flex treats the newline from TEST1\n as a newline at the beginning of a line
(i.e., empty line). Flex executes this rule:


^\n { action }


Such an ingenious problem!


Comments?


/Roger


[1] https://epaperpress.com/lexandyacc/download/flex.pdf
[This is a good example of the reasons you don't want your lexer to be too clever. -John]

Return to the comp.compilers page.
Search the comp.compilers archives again.