Dennis Kubes wrote:
> If you are using more than one index then dedup will not work across
> indexes.

This is incorrect. DeleteDuplicates works just fine with multiple
indexes, assuming you process all indexes in the same run of
DeleteDuplicates, so that it has a global view of all input indexes.

 A single index should dedup correctly unless the pages are not
> exact duplicates but near duplicates. The dedup process works on url
> and byte hash. If the content is even 1 byte different, it doesn't work.

This depends on the implementation of Signature. Indeed, the default
MD5HashSignature works this way.

>
> Near duplicate detection is another set of algorithms that haven't been
> implemented in Nutch yet.

Well, the existing TextProfileSignature can be used as a form of (crude)
near-duplicate detection, precisely because it is tolerant to small
changes in the input text.


--
Best regards,
Andrzej Bialecki   <><
___. ___ ___ ___ _ _  __________________________________
[__ || __|__/|__||\/| Information Retrieval, Semantic Web
___|||__|| \| || | Embedded Unix, System Integration
http://www.sigram.com Contact: info at sigram dot com