Re: Languages with optional spaces

awanderin <awanderin@gmail.com>
Wed, 26 Feb 2020 10:03:11 -0700

          From comp.compilers

Related articles
Languages with optional spaces maury.markowitz@gmail.com (Maury Markowitz) (2020-02-19)
Re: Languages with optional spaces awanderin@gmail.com (Jerry) (2020-02-20)
Re: Languages with optional spaces drikosev@gmail.com (Ev. Drikos) (2020-02-23)
Re: Languages with optional spaces maury.markowitz@gmail.com (Maury Markowitz) (2020-02-25)
Re: Languages with optional spaces maury.markowitz@gmail.com (Maury Markowitz) (2020-02-25)
Re: Languages with optional spaces martin@gkc.org.uk (Martin Ward) (2020-02-25)
Re: Languages with optional spaces 493-878-3164@kylheku.com (Kaz Kylheku) (2020-02-26)
Re: Languages with optional spaces awanderin@gmail.com (awanderin) (2020-02-26)
Re: Languages with optional spaces drikosev@gmail.com (Ev. Drikos) (2020-02-28)
Re: Languages with optional spaces christopher.f.clark@compiler-resources.com (Christopher F Clark) (2020-02-29)
Re: Languages with optional spaces drikosev@gmail.com (Ev. Drikos) (2020-02-29)
Re: Languages with optional spaces DrDiettrich1@netscape.net (Hans-Peter Diettrich) (2020-03-01)
Re: Languages with optional spaces christopher.f.clark@compiler-resources.com (Christopher F Clark) (2020-03-01)
Re: Languages with optional spaces drikosev@gmail.com (Ev. Drikos) (2020-03-01)
[7 later articles]
| List of all articles for this month |

From: awanderin <awanderin@gmail.com>
Newsgroups: comp.compilers
Date: Wed, 26 Feb 2020 10:03:11 -0700
Organization: A noiseless patient Spider
References: 20-02-015 20-02-016 20-02-019
Injection-Info: gal.iecc.com; posting-host="news.iecc.com:2001:470:1f07:1126:0:676f:7373:6970"; logging-data="4585"; mail-complaints-to="abuse@iecc.com"
Keywords: lex, Basic, history
Posted-Date: 27 Feb 2020 17:34:05 EST

Maury Markowitz <maury.markowitz@gmail.com> writes:


> Unless I'm mistaken, Integer BASIC is a "deep tokenizer", so is it not
> the case the spaces exist only at entry time? IE, if one types
> 'PRINT"HELLO"' that will actually be LISTed as 'PRINT "HELLO"?
> [That is my recollection. Storing tokens takes less space than
> storing the input text and on those tiny little
> micros, every byte counted. -John]


In the Apple II Microsoft BASIC, yes, the tokenizer removed spaces
except within strings. On the Commodore machines, their version of
Microsoft BASIC did not purge the spaces. This allowed the programmer
to "indent" the code (after the line number). The tokens, like PRINT,
FOR, INPUT, etc, are stored as one byte in all the Microsoft BASICs.
The Apple II Integer BASIC also tokenizes to one byte and removes
spaces, but it did more syntax checking at entry time, so it was more a
lexer + parser than a lexer like Microsoft BASIC.


--
--
Jerry awanderin at gmail dot com


Post a followup to this message

Return to the comp.compilers page.
Search the comp.compilers archives again.