Re: lex - how to get current file position of parser file

On 2015-09-01, Jens Kallup <jkallup@web.de> wrote:
> Hello,
>
> I used flex, and bison under Linux. But how can I handle multiple
> input files? When I follow the examples of the documentation, I fail
> to handle tokens after include file.
>
> Example:
>
> PRINT "Before"
> SET PROCEDURE TO testproc
> PRINT "After"


> this is a abstract language, which does (should) open a file called
> "testproc".


I think that in a dBase-type language, SET PROCEDURE doesn't necessarily have
to open the file at parse time. According to current documentation that
I'm able to find, it's actually become a complex module-loading command, and
not parse-time textual inclusion. It can load compiled files, not only text,
and cause non-persistent files to be unloaded. Loaded files have a refcount so
that if you invoke SET PROCEDURE on some file N times, you have to CLOSE
PROCEDURE that many times before it is unloaded.


> But I can not handle line 3.
> I try to "ftell" the position after testproc,
> but get the size of the parser file.
>
> So, now I have no glue, how to fix that. It would be great, if
> someone have a idea, and maybe a short code snippet.
>
> Thanks
> Jens
> [Flex buffers its input, so you can't just do an ftell(). Use the
> yy_create_buffer() and related routines. If you have a copy of my
> book "flex & bison" there's an example of nested include files in
> chapter 2. -John]


The benefit of yy_create_buffer et al is that you maintain the illusion that
there is just a single stream of tokens, though multiple files are being
shuffled around by the lexer, which maintains the "include stack". This
multiple stream of tokens is handled by a single parse job (one big yyparse).


Another approach (if you can design the language that way) is to specify
reentrant parser and lexer. This lets you issue a new, recursive call to
yyparse to handle the loaded/included file. The call has its own instance
of the lexer, which has no interaction with the suspended lexer of
the parent. If your parser builds an abstract syntax tree, then the phrase
structure rule for the include construct pulls out the AST from the nested
yyparse job, and integrates it into the surrounding AST. Of course, this can't
be used if you're implementing "dumb" textual inclusion, such that the syntax
matched by a single phrase structure rule can span across multiple files, and
even start in one file and end in another.

Return to the comp.compilers page.
Search the comp.compilers archives again.