Compilers for supercomputing or the opposite ?

wenhelt@free.fr
Fri, 05 Oct 2007 20:26:24 +0200

          From comp.compilers
Related articles
Compilers for supercomputing or the opposite ? wenhelt@free.fr (2007-10-05)
Re: Compilers for supercomputing or the opposite ? DrDiettrich1@aol.com (Hans-Peter Diettrich) (2007-10-06)
Re: Compilers for supercomputing or the opposite ? gah@ugcs.caltech.edu (glen herrmannsfeldt) (2007-10-06)
| List of all articles for this month |
From: wenhelt@free.fr
Newsgroups: comp.compilers
Date: Fri, 05 Oct 2007 20:26:24 +0200
Organization: Compilers Central
Keywords: symbols
Posted-Date: 05 Oct 2007 13:55:57 EDT

Although compiler code for supercomputing is commonplace,
supercomputing FOR compilers is coming.


For an EDA (Electronic Design Automation) project, we are facing
symbol tables over 500 million entries.


RAM of desktop machines is too short to gasp the name strings. Hash
tables are out.


Disk storage is the solution, but random disk access is desesperately
slow. Forget b-trees, indexes and databases.


Symbol lookups can be eliminated using inverted index on symbols.
Random disk access can be eliminated moving to sequential disk access.


Web search engines are pretty efficient building inverted index.
Thanks to supercomputing on desktop machines.


With MapReduce framework, reverting index is implemented in a page of
code and two hours. Two more hours are enough to revert index 10 giga
symbols on 24 machines.


How ?


http://code.google.com/edu/content/submissions/mapreduce/listing.html


Pascal HANNHART
Senior compiler developper


Post a followup to this message

Return to the comp.compilers page.
Search the comp.compilers archives again.