Re: Applesoft tokenization phases?

"Ev. Drikos" <>
Wed, 18 Mar 2020 00:14:51 +0200

          From comp.compilers

Related articles
Applesoft tokenization phases? (Ev. Drikos) (2020-03-12)
Re: Applesoft tokenization phases? (George Neuner) (2020-03-13)
Re: Applesoft tokenization phases? (awanderin) (2020-03-16)
Re: Applesoft tokenization phases? (Ev. Drikos) (2020-03-18)
Re: Applesoft tokenization phases? (Christopher F Clark) (2020-03-20)
Re: Applesoft tokenization phases? (Martin Ward) (2020-03-21)
| List of all articles for this month |

From: "Ev. Drikos" <>
Newsgroups: comp.compilers
Date: Wed, 18 Mar 2020 00:14:51 +0200
Organization: NNTP Server
References: 20-03-013 20-03-016
Injection-Info:; posting-host=""; logging-data="94207"; mail-complaints-to=""
Keywords: Basic, history
Posted-Date: 19 Mar 2020 17:44:31 EDT
Content-Language: en-US

On 16/03/2020 08:07, awanderin wrote:
> "Ev. Drikos" <> writes:
>> ...
>> Yet, I've found ie a program at "" [2] that contains:
>> 110 DIFF = ABS(A(I)-N)
> If you type that into Applesoft BASIC, it parses it as:
> 110 D IF F = ABS (A(I) - N)
> The spaces are how Applesoft lists it...

Another vague point or simply a point where I'm not really sure that I
translate properly the manual are the reserved keywords before a certain
delimiter. Likely an Applesoft parser must reject this valid UK101 code:

20 END

>> Also, an online AppleSoft simulator at [3] accepts ie both
>> DIFF and FEND as valid variable names.
> It is doing things differently than actual Applesoft.
>> As it seems, this issue can affect a design choice for the tokenization
>> phases of an Applesoft front-end. Is the manual just informative or the
>> online simulator does not accept (precisely) the particular dialect?
> The latter; the simulator accepts a different dialect.
> --

I've also read your comment for spacing rules on the Commodore machines.



IMHO, if spaces are important then a Lexer can be simpler. One can build
one (L1) with a lexer generator that supports intersection/negation and
just re-scan the DATA statements.

Due to complex spacing rules of AppleSoft II, one could scan in advance:
1. DATA statements by preserving spaces in literals & strings, and
        1.1 Strings & Comments along with the Keyword AT
2. All other Keywords once the remaining spaces have been skipped

Thereafter one has to scan just few more tokens, ie names & delimiters,
yet this task is too simple for a generated Lexer (L2).

With a proprietary tool (Syntaxis) that supports cascaded scanners, I
could model a solution that supports both forms (space optional or not)
by combining 1, 2, and then for space efficiency just reused L1 instead
of L2 for the remaining tokens. The tool built parses several examples
but in any case the implementation is too fresh to be considered stable.

Obviously one could hand code a lexer by following the above 3 phases.

Ev. Drikos

Post a followup to this message

Return to the comp.compilers page.
Search the comp.compilers archives again.