Re: lexer speed, was Bison

Hans-Peter Diettrich <>
Mon, 20 Aug 2012 01:01:53 +0100

          From comp.compilers

Related articles
=?UTF-8?Q?Bison_determinis=E2=80=8Btic_LALR=281=29_parser_for_Java=2FC (2012-08-17)
Re: Bison =?UTF-8?B?ZGV0ZXJtaW5pc+KAi3RpYyBMQUxSKDEpIHBhcnNlciBm?= =?U (Hans-Peter Diettrich) (2012-08-18)
Re: lexer speed, was Bison (Hans-Peter Diettrich) (2012-08-20)
Re: lexer speed, was Bison (Hans-Peter Diettrich) (2012-08-20)
Re: lexer speed, was Bison (BGB) (2012-08-20)
Re: lexer speed, was Bison (Hans-Peter Diettrich) (2012-08-21)
Re: lexer speed, was Bison (BartC) (2012-08-21)
| List of all articles for this month |

DKIM-Signature: v=1; a=rsa-sha256; c=simple;; h=cc:from:subject:date:sender:message-id:references:mime-version:content-type:content-transfer-encoding:vbr-info; s=eba9.50317a1a.k1208;; bh=F1pr3CwJ6XsJziyWgJJKRJ8NZ2tU5wvQ6+rdvM8gYBY=; b=IxGNLU86IRiE045ANqz9niQ13z37MDAKmGxndiYrzXfqJNn9v9ynxeubCutI6/rwK5w/Y78L1T45gRsJMgGwQZSYV8ac0h2mo15K1i4FfuoT7vbTWPxXTOqCtGoQ5n0QFXHoQJZdRf9nM+DJKz+NbIbveUGbczwPdRXWleY8WD4=
VBR-Info:; mc=all;
From: Hans-Peter Diettrich <>
Newsgroups: comp.compilers
Date: Mon, 20 Aug 2012 01:01:53 +0100
Organization: Compilers Central
References: 12-08-005 12-08-006
Keywords: parse, performance
Posted-Date: 19 Aug 2012 19:43:21 EDT

> [Compilers spend a lot of time in the lexer, because that's the only
> phase that has to look at the input one character at a time. -John]

When the source code resides in a memory buffer, the time for reading
e.g. the characters of an identifier (in the lexer) is neglectable vs.
the time spent in lookup and entering the identifier into a symbol table
(in the parser).

Even if a lexer reads single characters from a file, most OSs maintain
their own file buffer, so that little overhead is added over the
program-buffered solution.

I really would like to see some current benchmarks about the behaviour
of current compilers and systems.

[The benchmarks I did were a while ago, but they showed a large
fraction of time in the lexer. I wouldn't disagree that building the
symbol table is slow, but figure out some estimate of the ratio of
the number of characters in a source file to the number of tokens,
and that is a rough estimate of how much slower the lexer will be
than the parser. I agree that some current analyses would be useful.

Post a followup to this message

Return to the comp.compilers page.
Search the comp.compilers archives again.