Java Mailing List Archive

http://www.java2.5341.com/

Home » nutch-user.lucene »

Preferred nutch cluster network topology ?

brainstorm

2008-07-03


Author LoginPost Reply
Regarding real world nutch clusters (>10 nodes) what's the approach
you follow to maximise fetches throughput ?

For instance, my guess is that the "classical" number-crunching (HPC)
scientific network cluster topology (intra-cluster private network
plus 1 head node with "outside world" connection), it's suboptimal in
a nutch deployment: network bottleneck in head node while crawling
inet.

So what do you suggest in that matter ?

Thanks in advance !
©2008 java2.5341.com - Jax Systems, LLC, U.S.A.