|Compilers for supercomputing or the opposite ? firstname.lastname@example.org (2007-10-05)|
|Re: Compilers for supercomputing or the opposite ? DrDiettrich1@aol.com (Hans-Peter Diettrich) (2007-10-06)|
|Re: Compilers for supercomputing or the opposite ? email@example.com (glen herrmannsfeldt) (2007-10-06)|
|Date:||Fri, 05 Oct 2007 20:26:24 +0200|
Although compiler code for supercomputing is commonplace,
supercomputing FOR compilers is coming.
For an EDA (Electronic Design Automation) project, we are facing
symbol tables over 500 million entries.
RAM of desktop machines is too short to gasp the name strings. Hash
tables are out.
Disk storage is the solution, but random disk access is desesperately
slow. Forget b-trees, indexes and databases.
Symbol lookups can be eliminated using inverted index on symbols.
Random disk access can be eliminated moving to sequential disk access.
Web search engines are pretty efficient building inverted index.
Thanks to supercomputing on desktop machines.
With MapReduce framework, reverting index is implemented in a page of
code and two hours. Two more hours are enough to revert index 10 giga
symbols on 24 machines.
Senior compiler developper
Return to the
Search the comp.compilers archives again.