when I tested crawling a web deeply (-depth 100 -topN 1000), the job always failed with the exception below

Exception in thread "main" java.io.IOException: Job failed!
    at org.apache.hadoop.mapred.JobClient.runJob (JobClient.java:894)
    at org.apache.nutch.crawl.Generator.generate (Generator.java:456)
    at org.apache.nutch.crawl.Generator.generate (Generator.java:393)
    at org.apache.nutch.crawl.Crawl.main (Crawl.java:116)

log shows the following:
java.lang.OutOfMemoryError: PermGen space
    at java.lang.ClassLoader.defineClass1(Native Method)
    at java.lang.ClassLoader.defineClass (ClassLoader.java:620)
    at java.security.SecureClassLoader.defineClass (SecureClassLoader.java:124)
    at java.net.URLClassLoader.defineClass (URLClassLoader.java:260)
    at java.net.URLClassLoader.access$000(URLClassLoader.java:56)
    at java.net.URLClassLoader$1.run(URLClassLoader.java:195)
    at java.security.AccessController.doPrivileged(Native Method)
    at java.net.URLClassLoader.findClass (URLClassLoader.java:188)
    at java.lang.ClassLoader.loadClass (ClassLoader.java:306)
    at java.lang.ClassLoader.loadClass (ClassLoader.java:251)
    at org.apache.nutch.plugin.Extension.getExtensionInstance (Extension.java:156)
    at org.apache.nutch.net.URLNormalizers.getURLNormalizers (URLNormalizers.java:170)
    at org.apache.nutch.net.URLNormalizers.(URLNormalizers.java:128)
    at org.apache.nutch.crawl.Generator$Selector.configure(Generator.java:109)
    at org.apache.hadoop.util.ReflectionUtils.setConf (ReflectionUtils.java:58)
    at org.apache.hadoop.util.ReflectionUtils.newInstance (ReflectionUtils.java:82)
    at org.apache.hadoop.mapred.ReduceTask.run (ReduceTask.java:250)
    at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:164)



Is this is memory problem? I set the hadoop parameter

mapred.child.java.opts
-Xmx512m


Any idea why this keeps happening? Many thanks!