Re: replacements for LEX: flex, ... , Rex

Josef Grosch describes Rex, a scanner generator similar to lex,
and gives some timings comparing its perfomance with lex's:


> Scanners generated by Rex are more than 4 times faster and up to 5
> times smaller than LEX-generated ones. It is possible to reach a speed
> of 180,000 to 195,000 lines per minute on a MC 68020 processor (includ-
> ing input from file)....
> .... I would like to hear some numbers from "flex".


Here they are for a C tokenizer which includes keywords.


Generation times:


lex 83.0 secs
flex 3.9
flex -cfe 7.1 # uncompressed table, equivalence classes
flex -cf 15.0 # uncompressed table, no equivalence classes


Scanner object file sizes:


lex 41.0K bytes
flex 9.4K
flex -cfe 49.6K
flex -cf 126.5K


Running times on a 28,088 line input (685K characters):


lex 29.8 secs
flex 19.3
flex -cfe 9.0
flex -cf 7.8


The timings were made on a Sun 3/60. All times are user + system CPU time,
and don't include hashing of identifiers.


Summary:


        For about the same sized scanner, you get a factor of 3 in performance.
        For a 30% faster scanner, you get a scanner 1/4th the size, and it's
generated in 1/20th the time.
        For a scanner that's 3 times larger, you get a factor of 3.8 in
performance.


Rex sounds very good - taken separately, various options of flex perform in
about its league, but flex can't match its combined performance. This is
due in part to flex's compressed scanners not having been optimized for
speed nearly as much as the full table scanners.


What is Rex's availability?


Vern


Vern Paxson vern@lbl-csam.arpa
Real Time Systems ucbvax!lbl-csam.arpa!vern
Lawrence Berkeley Laboratory (415) 486-6411
--

Return to the comp.compilers page.
Search the comp.compilers archives again.