Re: lex/flex for perl

On 28 Jul 1998 11:29:30 -0400, David Maclean <david@cm.co.za> wrote:


>Can anyone tell me if there is a lex/flex for perl. There is byacc for
>perl but have not found lex for perl.


If you're interested in writing compilers in Perl, take a look at the
"RecDescent" module distributed through CPAN. It's really quite good;
it uses Perl's dynamic facilities to allow you to do things on the fly
that you could never do in yacc or its cousins. You can modify
grammars on the fly in very intuitive ways, for instance.


And yes, John is right: you don't need lex for Perl, as it has lex
sort of built in, if you structure things right.


Be aware, though, that however you do it in Perl, you are not going to
break any speed records. But that's okay for many (maybe most)
applications.


Another thing, David: if you don't want to learn RecDescent, you can
always write recursive descent parsers by hand in Perl almost as fast
as you can write down the EBNF, once you know how. This was a
standard skill of the olden days (early 70's was when I learned it)
but it seems to be a disappearing art, which is too bad, as it can be
a very useful wrench for your toolkit. One function per non-terminal,
use loops for closures and if's for alternatives, and so forth. You
get LL(k) grammars that way, with tightly-controlled lookahead,
essentially the ANTLR predicate approach but without the tools.


There is a whole different way of doing things, too, if what you want
is a little domain-specific language, and that is to abandon lexing
and parsing in the traditional sense and instead go to a line-oriented
approach that takes maximal advantage of Perl's strengths. I've
written three little "languages" this way in the past year, and it's
worked out very well. The idea is to restrict your language to
contructs that are easily identified by Perl's regexs, and then read
the source in a line at a time, and run each line through a series of
regex conditions that identifies and handles that kind of line. In
other words, you give up free-form coding, and you require whitespace
in many places, and you insist that most constructs start at the
beginning of a line. You read in the next line with


line: while(<>) {
chop;


and you skip whitespace and comments with


next line if /^\s*$/;
next line if /^\s*#/;


and you maybe use split to divide the input into words (or maybe not,
depending on your language) and you keep global state variables around
telling you if you're inside a function or conditional or loop, and so
on.


Yes, yes, I know ... it's a hack. But you can write and debug one of
these things in a day, and then you can rewrite it later in less time
than that. You can make it pretty robust, and with the liberal use of
comments, it's not really all that bad. Perl is really quite an
extraordinary language for the language/tools programmer, in my
opinion.


Dwight
[I've written a zillion little languages in perl. Great stuff. -John]
--

Return to the comp.compilers page.
Search the comp.compilers archives again.