Error in simple Hadoop Map Reduce
我尝试运行hadoop地图来减少字数统计问题。据我所知,我正确设置了HDFS以及所有内容。
执行时,
在尝试运行一个简单的字数统计问题时出现此错误。我是Hadoop的新手。任何解决方法的帮助将不胜感激。
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54 55 56 57 58 59 60 61 62 63 64 65 66 67 68 69 70 71 72 73 74 75 76 77 | 13/06/13 20:21:17 INFO input.FileInputFormat: Total input paths to process : 4 13/06/13 20:21:17 INFO util.NativeCodeLoader: Loaded the native-hadoop library 13/06/13 20:21:17 WARN snappy.LoadSnappy: Snappy native library not loaded 13/06/13 20:21:17 INFO mapred.JobClient: Running job: job_201306131940_0007 13/06/13 20:21:18 INFO mapred.JobClient: map 0% reduce 0% 13/06/13 20:21:21 INFO mapred.JobClient: Task Id : attempt_201306131940_0007_m_000005_0, Status : FAILED java.lang.Throwable: Child Error at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271) Caused by: java.io.IOException: Creation of symlink from /home/vraju/Desktop/hadoop/libexec/../logs/userlogs/job_201306131940_0007/attempt_201306131940_0007_m_000005_0 to /app/hadoop/tmp/mapred/local/userlogs/job_201306131940_0007/attempt_201306131940_0007_m_000005_0 failed. at org.apache.hadoop.mapred.TaskLog.createTaskAttemptLogDir(TaskLog.java:110) at org.apache.hadoop.mapred.DefaultTaskController.createLogDir(DefaultTaskController.java:71) at org.apache.hadoop.mapred.TaskRunner.prepareLogFiles(TaskRunner.java:316) at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:228) 13/06/13 20:21:21 WARN mapred.JobClient: Error reading task outputhttp://ubuntu:50060/tasklog?plaintext=true&attemptid=attempt_201306131940_0007_m_000005_0&filter=stdout 13/06/13 20:21:21 WARN mapred.JobClient: Error reading task outputhttp://ubuntu:50060/tasklog?plaintext=true&attemptid=attempt_201306131940_0007_m_000005_0&filter=stderr 13/06/13 20:21:24 INFO mapred.JobClient: Task Id : attempt_201306131940_0007_m_000005_1, Status : FAILED java.lang.Throwable: Child Error at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271) Caused by: java.io.IOException: Creation of symlink from /home/vraju/Desktop/hadoop/libexec/../logs/userlogs/job_201306131940_0007/attempt_201306131940_0007_m_000005_1 to /app/hadoop/tmp/mapred/local/userlogs/job_201306131940_0007/attempt_201306131940_0007_m_000005_1 failed. at org.apache.hadoop.mapred.TaskLog.createTaskAttemptLogDir(TaskLog.java:110) at org.apache.hadoop.mapred.DefaultTaskController.createLogDir(DefaultTaskController.java:71) at org.apache.hadoop.mapred.TaskRunner.prepareLogFiles(TaskRunner.java:316) at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:228) 13/06/13 20:21:25 WARN mapred.JobClient: Error reading task outputhttp://ubuntu:50060/tasklog?plaintext=true&attemptid=attempt_201306131940_0007_m_000005_1&filter=stdout 13/06/13 20:21:25 WARN mapred.JobClient: Error reading task outputhttp://ubuntu:50060/tasklog?plaintext=true&attemptid=attempt_201306131940_0007_m_000005_1&filter=stderr 13/06/13 20:21:28 INFO mapred.JobClient: Task Id : attempt_201306131940_0007_m_000005_2, Status : FAILED java.lang.Throwable: Child Error at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271) Caused by: java.io.IOException: Creation of symlink from /home/vraju/Desktop/hadoop/libexec/../logs/userlogs/job_201306131940_0007/attempt_201306131940_0007_m_000005_2 to /app/hadoop/tmp/mapred/local/userlogs/job_201306131940_0007/attempt_201306131940_0007_m_000005_2 failed. at org.apache.hadoop.mapred.TaskLog.createTaskAttemptLogDir(TaskLog.java:110) at org.apache.hadoop.mapred.DefaultTaskController.createLogDir(DefaultTaskController.java:71) at org.apache.hadoop.mapred.TaskRunner.prepareLogFiles(TaskRunner.java:316) at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:228) 13/06/13 20:21:28 WARN mapred.JobClient: Error reading task outputhttp://ubuntu:50060/tasklog?plaintext=true&attemptid=attempt_201306131940_0007_m_000005_2&filter=stdout 13/06/13 20:21:28 WARN mapred.JobClient: Error reading task outputhttp://ubuntu:50060/tasklog?plaintext=true&attemptid=attempt_201306131940_0007_m_000005_2&filter=stderr 13/06/13 20:21:34 INFO mapred.JobClient: Task Id : attempt_201306131940_0007_m_000004_0, Status : FAILED java.lang.Throwable: Child Error at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271) Caused by: java.io.IOException: Creation of symlink from /home/vraju/Desktop/hadoop/libexec/../logs/userlogs/job_201306131940_0007/attempt_201306131940_0007_m_000004_0 to /app/hadoop/tmp/mapred/local/userlogs/job_201306131940_0007/attempt_201306131940_0007_m_000004_0 failed. at org.apache.hadoop.mapred.TaskLog.createTaskAttemptLogDir(TaskLog.java:110) at org.apache.hadoop.mapred.DefaultTaskController.createLogDir(DefaultTaskController.java:71) at org.apache.hadoop.mapred.TaskRunner.prepareLogFiles(TaskRunner.java:316) at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:228) 13/06/13 20:21:34 WARN mapred.JobClient: Error reading task outputhttp://ubuntu:50060/tasklog?plaintext=true&attemptid=attempt_201306131940_0007_m_000004_0&filter=stdout 13/06/13 20:21:34 WARN mapred.JobClient: Error reading task outputhttp://ubuntu:50060/tasklog?plaintext=true&attemptid=attempt_201306131940_0007_m_000004_0&filter=stderr 13/06/13 20:21:37 INFO mapred.JobClient: Task Id : attempt_201306131940_0007_m_000004_1, Status : FAILED java.lang.Throwable: Child Error at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271) Caused by: java.io.IOException: Creation of symlink from /home/vraju/Desktop/hadoop/libexec/../logs/userlogs/job_201306131940_0007/attempt_201306131940_0007_m_000004_1 to /app/hadoop/tmp/mapred/local/userlogs/job_201306131940_0007/attempt_201306131940_0007_m_000004_1 failed. at org.apache.hadoop.mapred.TaskLog.createTaskAttemptLogDir(TaskLog.java:110) at org.apache.hadoop.mapred.DefaultTaskController.createLogDir(DefaultTaskController.java:71) at org.apache.hadoop.mapred.TaskRunner.prepareLogFiles(TaskRunner.java:316) at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:228) 13/06/13 20:21:37 WARN mapred.JobClient: Error reading task outputhttp://ubuntu:50060/tasklog?plaintext=true&attemptid=attempt_201306131940_0007_m_000004_1&filter=stdout 13/06/13 20:21:37 WARN mapred.JobClient: Error reading task outputhttp://ubuntu:50060/tasklog?plaintext=true&attemptid=attempt_201306131940_0007_m_000004_1&filter=stderr 13/06/13 20:21:40 INFO mapred.JobClient: Task Id : attempt_201306131940_0007_m_000004_2, Status : FAILED java.lang.Throwable: Child Error at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271) Caused by: java.io.IOException: Creation of symlink from /home/vraju/Desktop/hadoop/libexec/../logs/userlogs/job_201306131940_0007/attempt_201306131940_0007_m_000004_2 to /app/hadoop/tmp/mapred/local/userlogs/job_201306131940_0007/attempt_201306131940_0007_m_000004_2 failed. at org.apache.hadoop.mapred.TaskLog.createTaskAttemptLogDir(TaskLog.java:110) at org.apache.hadoop.mapred.DefaultTaskController.createLogDir(DefaultTaskController.java:71) at org.apache.hadoop.mapred.TaskRunner.prepareLogFiles(TaskRunner.java:316) at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:228) 13/06/13 20:21:40 WARN mapred.JobClient: Error reading task outputhttp://ubuntu:50060/tasklog?plaintext=true&attemptid=attempt_201306131940_0007_m_000004_2&filter=stdout 13/06/13 20:21:40 WARN mapred.JobClient: Error reading task outputhttp://ubuntu:50060/tasklog?plaintext=true&attemptid=attempt_201306131940_0007_m_000004_2&filter=stderr 13/06/13 20:21:43 INFO mapred.JobClient: Job complete: job_201306131940_0007 13/06/13 20:21:43 INFO mapred.JobClient: Counters: 4 13/06/13 20:21:43 INFO mapred.JobClient: Job Counters 13/06/13 20:21:43 INFO mapred.JobClient: SLOTS_MILLIS_MAPS=24100 13/06/13 20:21:43 INFO mapred.JobClient: Total time spent by all reduces waiting after reserving slots (ms)=0 13/06/13 20:21:43 INFO mapred.JobClient: Total time spent by all maps waiting after reserving slots (ms)=0 13/06/13 20:21:43 INFO mapred.JobClient: SLOTS_MILLIS_REDUCES=0 |
当您获得
1 2 | java.lang.Throwable: Child Error at org.apache.hadoop.mapred.TaskRunner.run(TaskRunner.java:271) |
似乎为任务跟踪器分配的内存大于节点的实际内存。
您可以在此处获得更多信息
在尝试执行相同的操作时,我遇到了完全相同的错误-对简单文本执行字数统计。
就我而言,我解决了更改Hadoop依赖的JDK版本的问题。我正在使用Hadoop 1.2.1,并且在网上发现它的首选Java版本是1.6
我遇到了这个问题,但这是程序包名称问题。
我正在使用Maven进行编译。在pom.xml中,我们有
我所有的软件包都必须在GROUPNAME内,例如GROUPNAME.newpackage.ClassName
然后,它起作用了。
我遇到了一整天都解决的问题,直到我在线找到一个解决方案,即VirtualBox的共享目录中不支持创建符号链接的操作。尝试将hadoop目录移至/ usr / local能否解决您的问题。