在沙箱Hadoop(Cloudera5.1/Hortonworks2.1)的R(build 1060)中是否有类似的问题?这似乎是新R/Hadoop的问题,因为在CDH5.0上它可以工作.
码:
Sys.setenv(HADOOP_CMD="/usr/bin/hadoop") Sys.setenv(HADOOP_STREAMING="/usr/lib/hadoop-mapreduce/hadoop-streaming.jar") Sys.setenv(JAVA_HOME="/usr/java/jdk1.7.0_55-cloudera") library(rhdfs) library(rmr2) hdfs.init() ## space and word delimiter map <- function(k,lines) { words.list <- strsplit(lines, '\\s') words <- unlist(words.list) return( keyval(words, 1) ) } reduce <- function(word, counts) { keyval(word, sum(counts)) } wordcount <- function (input, output=NULL) { mapreduce(input=input, output=output, input.format="text", map=map, reduce=reduce) } ## variables hdfs.root <- '/user/cloudera' hdfs.data <- file.path(hdfs.root, 'scenario_1') hdfs.out <- file.path(hdfs.root, 'out') ## run mapreduce job ##out <- wordcount(hdfs.data, hdfs.out) system.time(out <- wordcount(hdfs.data, hdfs.out))
错误:
> system.time(out <- wordcount(hdfs.data, hdfs.out)) packageJobJar: [] [/usr/lib/hadoop-mapreduce/hadoop-streaming-2.3.0-cdh5.1.0.jar] /tmp/streamjob8497498354509963133.jar tmpDir=null 14/09/17 01:49:38 INFO client.RMProxy: Connecting to ResourceManager at quickstart.cloudera/127.0.0.1:8032 14/09/17 01:49:38 INFO client.RMProxy: Connecting to ResourceManager at quickstart.cloudera/127.0.0.1:8032 14/09/17 01:49:39 INFO mapred.FileInputFormat: Total input paths to process : 1 14/09/17 01:49:39 INFO mapreduce.JobSubmitter: number of splits:2 14/09/17 01:49:39 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_1410940439997_0001 14/09/17 01:49:40 INFO impl.YarnClientImpl: Submitted application application_1410940439997_0001 14/09/17 01:49:40 INFO mapreduce.Job: The url to track the job: http://quickstart.cloudera:8088/proxy/application_1410940439997_0001/ 14/09/17 01:49:40 INFO mapreduce.Job: Running job: job_1410940439997_0001 14/09/17 01:49:54 INFO mapreduce.Job: Job job_1410940439997_0001 running in uber mode : false 14/09/17 01:49:54 INFO mapreduce.Job: map 100% reduce 100% 14/09/17 01:49:55 INFO mapreduce.Job: Job job_1410940439997_0001 failed with state KILLED due to: MAP capability required is more than the supported max container capability in the cluster. Killing the Job. mapResourceReqt: 4096 maxContainerCapability:1024 Job received Kill while in RUNNING state. REDUCE capability required is more than the supported max container capability in the cluster. Killing the Job. **reduceResourceReqt: 4096 maxContainerCapability:1024** 14/09/17 01:49:55 INFO mapreduce.Job: Counters: 2 Job Counters Total time spent by all maps in occupied slots (ms)=0 Total time spent by all reduces in occupied slots (ms)=0 14/09/17 01:49:55 ERROR streaming.StreamJob: Job not Successful! Streaming Command Failed! Error in mr(map = map, reduce = reduce, combine = combine, vectorized.reduce, : hadoop streaming failed with error code 1 Timing stopped at: 3.681 0.695 20.43
似乎问题出在reduceResourceReqt:4096 maxContainerCapability:1024.我试图改变:yarn-site.xml,但它没有帮助.:(
请帮忙...