Re: Parsing HTML : I would appreciate advice

Jim escribió:
> The problem to solve.
>
> I have to parse millions of html documents, and return just the
> plaintext/bytes. Many of the html documents contain Japanese
> characters and so it will be necessary to read the codepage in the
> html header, so the bytes can be read properly.
>
> Ninety percent of the html documents are well formed, originally
> created by code. The rest of the documents are random html documents
> from the internet. I will be placing the plaintext in an SQL database
> and use full-text search.
>
> What type of parser/lexer is best for parsing html?
>
> Can you offer any links or existing libraries for doing this?
>
> I am most proficient in: C#, C++, C languages.


No need to write any code (well, almost). There are a lot of utilities
that can do what you want. A possiblity is to use 'xsltproc' with the
--html flag to apply an XSLT stlesheet that selects:


          <xsl:value-of select="/html/body" />


For not well formed pages, you can use 'Tidy' to fix them first.


And better ask in comp.text.xml or comp.infosystems.www.authoring.html.
They are probably more appropriate newsgroups for this subject.


Regards.
--
Manuel Collado - http://lml.ls.fi.upm.es/~mcollado

Return to the comp.compilers page.
Search the comp.compilers archives again.