Re: Applesoft tokenization phases?

Martin Ward <>
Sat, 21 Mar 2020 19:42:17 +0000

          From comp.compilers

Related articles
Applesoft tokenization phases? (Ev. Drikos) (2020-03-12)
Re: Applesoft tokenization phases? (George Neuner) (2020-03-13)
Re: Applesoft tokenization phases? (awanderin) (2020-03-16)
Re: Applesoft tokenization phases? (Ev. Drikos) (2020-03-18)
Re: Applesoft tokenization phases? (Christopher F Clark) (2020-03-20)
Re: Applesoft tokenization phases? (Martin Ward) (2020-03-21)
| List of all articles for this month |

From: Martin Ward <>
Newsgroups: comp.compilers
Date: Sat, 21 Mar 2020 19:42:17 +0000
Organization: Compilers Central
References: 20-03-013 20-03-016 20-03-017 20-03-022
Injection-Info:; posting-host=""; logging-data="66018"; mail-complaints-to=""
Keywords: Basic, lex, history
Posted-Date: 21 Mar 2020 16:50:24 EDT

On 17/03/20 22:14, Ev. Drikos wrote:
> Another vague point or simply a point where I'm not really sure that I
> translate properly the manual are the reserved keywords before a certain
> delimiter. Likely an Applesoft parser must reject this valid UK101 code:
> 10 X=SHIMEM:
> 20 END

For those less familiar with UK101 BASIC: this code is only
valid because "HIMEM" is not a keyword in UK101 BASIC.
UK101 does not allow keywords as part of a variable.

> There is a reason, BASIC compilers that remove spaces often restrict
> variables (like the original Basic version did) to a letter optionally
> followed by a digit (and a dollar sign to indicate strings). It makes
> the lexing much simpler, even without spaces.

Lexing is not the issue: the real reason for the restriction
is to make the symbol table much simpler. Each variable in the table
takes up a fixed amount of space. For the UK101, only the first two
characters of a variable are significant, so only two characters
are needed in the symbol table. "Lexing" a statement just consists
of replacing all keywords outside strings by special characters.
When the interpreted parses it, each statement either starts
with a keyword or is an assignment and starts with a variable.


Dr Martin Ward | Email: |
G.K.Chesterton site: | Erdos number: 4

Post a followup to this message

Return to the comp.compilers page.
Search the comp.compilers archives again.