Re: Languages with optional spaces

Maury Markowitz <maury.markowitz@gmail.com> writes:


> Unless I'm mistaken, Integer BASIC is a "deep tokenizer", so is it not
> the case the spaces exist only at entry time? IE, if one types
> 'PRINT"HELLO"' that will actually be LISTed as 'PRINT "HELLO"?
> [That is my recollection. Storing tokens takes less space than
> storing the input text and on those tiny little
> micros, every byte counted. -John]


In the Apple II Microsoft BASIC, yes, the tokenizer removed spaces
except within strings. On the Commodore machines, their version of
Microsoft BASIC did not purge the spaces. This allowed the programmer
to "indent" the code (after the line number). The tokens, like PRINT,
FOR, INPUT, etc, are stored as one byte in all the Microsoft BASICs.
The Apple II Integer BASIC also tokenizes to one byte and removes
spaces, but it did more syntax checking at entry time, so it was more a
lexer + parser than a lexer like Microsoft BASIC.


--
--
Jerry awanderin at gmail dot com

Return to the comp.compilers page.
Search the comp.compilers archives again.