Re: Applesoft tokenization phases?

Christopher F Clark <>
Fri, 20 Mar 2020 07:06:55 -0400

          From comp.compilers

Related articles
Applesoft tokenization phases? (Ev. Drikos) (2020-03-12)
Re: Applesoft tokenization phases? (George Neuner) (2020-03-13)
Re: Applesoft tokenization phases? (awanderin) (2020-03-16)
Re: Applesoft tokenization phases? (Ev. Drikos) (2020-03-18)
Re: Applesoft tokenization phases? (Christopher F Clark) (2020-03-20)
Re: Applesoft tokenization phases? (Martin Ward) (2020-03-21)
| List of all articles for this month |

From: Christopher F Clark <>
Newsgroups: comp.compilers
Date: Fri, 20 Mar 2020 07:06:55 -0400
Organization: Compilers Central
References: 20-03-013 20-03-016 20-03-017
Injection-Info:; posting-host=""; logging-data="86364"; mail-complaints-to=""
Keywords: Basic, history, lex
Posted-Date: 20 Mar 2020 11:46:44 EDT

Jerry Awanderin wrote:
> > Also, an online AppleSoft simulator at [3] accepts ie both
> > DIFF and FEND as valid variable names.
> It is doing things differently than actual Applesoft.
> > As it seems, this issue can affect a design choice for the tokenization
> > phases of an Applesoft front-end. Is the manual just informative or the
> > online simulator does not accept (precisely) the particular dialect?
> The latter; the simulator accepts a different dialect.

For situations like this, we added classes (and inheritance) to Yacc++
lexers and parsers, so that one could define a common subset and then
extend it to cover the cases that are unique or different dialects.
However, one can do similar things with lexer states, flag/switch
variables, or a host of other techniques. You just have to decide how
much mess you are willing to handle and how important supporting the
variations are.

There is a reason, BASIC compilers that remove spaces often restrict
variables (like the original Basic version did) to a letter optionally
followed by a digit (and a dollar sign to indicate strings). It makes
the lexing much simpler, even without spaces.

Chris Clark email:
Compiler Resources, Inc. Web Site:
23 Bailey Rd voice: (508) 435-5016
Berlin, MA 01503 USA twitter: @intel_chris

Post a followup to this message

Return to the comp.compilers page.
Search the comp.compilers archives again.