Re: how to push token back for Bison to consider it? (C)
15 May 2003 11:38:47 -0400

          From comp.compilers

Related articles
how to push token back for Bison to consider it? (Milan Durovic) (2003-04-13)
Re: how to push token back for Bison to consider it? (2003-05-06)
Re: how to push token back for Bison to consider it? (2003-05-15)
| List of all articles for this month |

From: (C)
Newsgroups: comp.compilers
Date: 15 May 2003 11:38:47 -0400
References: 03-04-038
Keywords: parse
Posted-Date: 15 May 2003 11:38:47 EDT

Milan Durovic <> wrote in message news:03-04-038...
> I want to do the following: when a certain grammar rule is matched I
> want to, in the action part, read all the tokens until a certain token
> is read, and then give the control back to the parser, who should
> continue with that token as the next token. In other words, I am doing
> something like:
> ruleBlah: nonterm1 nonterm2
> {
> while( yylex() != SOME_PARTICULAR_TOKEN )
> { do_something(); }
> }
> ruleBlah2: SOME_PARTICULAR_TOKEN nonterm3 nonterm4

A quick (and, in my option, dirty) method would be to put a wrapper
around yylex(); id est insert the following code into the top of your
parser ...

static int repeatToken = 0;

static int yylexWrapper( void ) {
        if( repeatToken != 0 ) return yylex();
                int temp = repeatToken;
                repeatToken = 0;
                return temp;
} }

#define yylex yylexWrapper /* monkey with the internals of bison */

Then add 'repeatToken = SOME_PARTICULAR_TOKEN;' to the ruleBlah processor.


If this is too slow you could try the following line noise :-)...

static int repeatToken = 0;
#define yylex() ((repeatToken?lex():repeatToken),repeatToken=0)

and in the scanner

#define yylex lex


The trouble with both these pieces of code is they may mess up if
someone decides to alter the internals of the code BISON generates,
particularly the mess of #define statements.

A more elegant way would be to implement your own scanner which caches
the last token, then implement a 'scan_goBack()' call to inform the
scanner to use the cached version, or implement a
'scan_readUntil( int token )' call to read upto your specified token.

This is the solution I chose when faced with a similar problem
a few months ago, and can be quite effective provided your scaned
tokens are fairly simple.

C 2003/5/8

[code examples are untested, but should work (hopefully)]

Post a followup to this message

Return to the comp.compilers page.
Search the comp.compilers archives again.