Re: is lex useful?

Robert.Corbett@Eng.Sun.COM (Robert Corbett) writes:


        [Concerning problems in scanning Fortran...]
|> A better example is


|> DO10I = expr1, expr2


|> Since the length of expr1 is bounded only by the number of characters
|> allowed in a statement, either a multipass lexer or practically
|> unbounded lookahead are needed.


|> Because Fortran limits the maximum size of a statement, a lexer for
|> Fortran can analyze any Fortran statement in constant time.


The finite maximum length of a statement also makes rescanning a lot
easier. Since you never need unlimited backtracking, you can simply
cache the whole statement in memory, and backtrack over it there.


|> [Right, thanks for the correction. In the DO10I example, note that just
|> looking ahead for a comma isn't sufficient. You have to look for a comma
|> not enclosed in parens, which lex can't do, because REs can't count. -John]


Even counting parentheses isn't sufficient. What about commas in e.g.:
Hollerith constants? (This works both ways: a parentheses in a
Hollerith constant doesn't inhibit the interpretation of a comma
later.)


I once had a bet with a person over the question. He claimed that the
best solution was to define some sort of predicate, and scan ahead to
avoid backtracking. For every predicate he suggested, however, I was
able to come up with an example statement for which it failed. (It's
been long enough since I've done any Fortran that I wouldn't want to
make the same bet today, but the possibility of using Hollerith
constants in an expression certainly requires a fair amount of
intelligence in any look-ahead scanner.)
--
James Kanze Tel.: (+33) 88 14 49 00 email: kanze@gabi-soft.fr
GABI Software, Sarl., 8 rue des Francs-Bourgeois, F-67000 Strasbourg, France
[What I did was to buffer up the whole statement, then make a pass over it
to extract quoted and hollerith constants, then another pass to decide if
it was an assignment-ish statement or not, and then lex and parse. -John]
--

Return to the comp.compilers page.
Search the comp.compilers archives again.