Re: Applesoft tokenization phases?

On 16/03/2020 08:07, awanderin wrote:
> "Ev. Drikos" <drikosev@gmail.com> writes:
>> ...
>> Yet, I've found ie a program at "hoist-point.com" [2] that contains:
>> 110 DIFF = ABS(A(I)-N)
>
> If you type that into Applesoft BASIC, it parses it as:
>
> 110 D IF F = ABS (A(I) - N)
>
> The spaces are how Applesoft lists it...
>


Another vague point or simply a point where I'm not really sure that I
translate properly the manual are the reserved keywords before a certain
delimiter. Likely an Applesoft parser must reject this valid UK101 code:


10 X=SHIMEM:
20 END




>> Also, an online AppleSoft simulator at calormen.com [3] accepts ie both
>> DIFF and FEND as valid variable names.
>
> It is doing things differently than actual Applesoft.
>
>> As it seems, this issue can affect a design choice for the tokenization
>> phases of an Applesoft front-end. Is the manual just informative or the
>> online simulator does not accept (precisely) the particular dialect?
>
> The latter; the simulator accepts a different dialect.
> --


I've also read your comment for spacing rules on the Commodore machines.


Thanks.




@everyone


IMHO, if spaces are important then a Lexer can be simpler. One can build
one (L1) with a lexer generator that supports intersection/negation and
just re-scan the DATA statements.


Due to complex spacing rules of AppleSoft II, one could scan in advance:
1. DATA statements by preserving spaces in literals & strings, and
        1.1 Strings & Comments along with the Keyword AT
2. All other Keywords once the remaining spaces have been skipped


Thereafter one has to scan just few more tokens, ie names & delimiters,
yet this task is too simple for a generated Lexer (L2).


With a proprietary tool (Syntaxis) that supports cascaded scanners, I
could model a solution that supports both forms (space optional or not)
by combining 1, 2, and then for space efficiency just reused L1 instead
of L2 for the remaining tokens. The tool built parses several examples
but in any case the implementation is too fresh to be considered stable.


Obviously one could hand code a lexer by following the above 3 phases.


Ev. Drikos

Return to the comp.compilers page.
Search the comp.compilers archives again.