Re: State-of-the-art algorithms for lexical analysis?

Yes, as our moderator explained. I was talking about things like
FORTRAN Hollerith strings, but more importantly network packets, where
they give the size of the "field" within a packet and then you simply
take that many characters (or bytes or bits or some other quanta) as
the "token". This is quite important for parsing "binary" data. And,
sometimes the numbers are text like I showed but in many protocols the
numbers are "binary" e.g. something like


\xAHabcdefghij where \xA is a single 8 bit character (octet) whose
bits are "0000 1010" (or maybe 4, 8 bit, characters -- 4 octets),
that represent a 32 integer).


And, as our moderator pointed out, this makes a terrible regular
expression, NFA, DFA, but it is actually quite easy in nearly any
programming language. You read the length in, convert it to an integer
and then loop reading that many characters from the input and call
that a "token".


Kind regards,
Chris


--
******************************************************************************
Chris Clark email: christopher.f.clark@compiler-resources.com
Compiler Resources, Inc. Web Site: http://world.std.com/~compres
23 Bailey Rd voice: (508) 435-5016
Berlin, MA 01503 USA twitter: @intel_chris
------------------------------------------------------------------------------
[Right. When I was writing Fortran lexers, Hollerith strings were among the
simplest of the kludges I had to use. -John]

Return to the comp.compilers page.
Search the comp.compilers archives again.