Re: flex scanner too huge, suggestions?

Troy Cauble <troy@bell-labs.com> wrote:
: I have a working flex/bison implementation to parse a text
: protocol. The problem is that the scanner is huge.


: Made with LFLAGS = -Cfe, size gives
: 42068(.text) + 24(.data) + 60(.bss) + 1687886(.rodata) = 1730038


: Made with LFLAGS = -Cem, size gives
: 42496(.text) + 24(.data) + 60(.bss) + 508326(.rodata) = 550906


: This is still pretty large for some of our target environments.


: I haven't tried to disable features to determine what makes
: it so huge, but the protocol is case-insensitive and context
: (position) sensitive. I addressed the context sensitivity
: with flex start states.




Thanks to all who replied here and via e-mail.


I thought I'd post what worked for future archive readers.


The big win for my flex scanner was rewriting INCLUSIVE start
states as EXCLUSIVE. I had about 50 start states with
1 to 5 rules each, to handle portions of the protocol.


Moving to EXTERNAL start states meant I had to add additional
rules for each start state (but a small subset of the INITIAL
rules). This resulted in more rules in the source file, but
much smaller tables. .rodata is less than half of what it was
at -Cem.


Given the table reduction, I'm *guessing* that 50 inclusive
start states generates 50 times the INITIAL (non start state)
rules. (Standard rules plus those in start state 1; standard
rules plus those in start state 2; ...)


Oddly enough, the oft-suggested "single identifier rule with
keyword lookup" actually increased my table size slightly
before I did rewrote the start states. I can't guess why.
(It wasn't the lookup function, I didn't even bother to
write that.)


-troy

Return to the comp.compilers page.
Search the comp.compilers archives again.