我在kerberized集群上运行Spark 1.1.0,HDP 2.1.我可以使用--master yarn-client成功运行spark-submit,并将结果正确写入HDFS,但是,该作业未显示在Hadoop All Applications页面上.我想使用--master yarn-cluster运行spark-submit但我继续收到此错误:
appDiagnostics: Application application_1417686359838_0012 failed 2 times due to AM Container for appattempt_1417686359838_0012_000002 exited with exitCode: -1000 due to: File does not exist: hdfs:///user/ /.sparkStaging/application_ _ /spark-assembly-1.1.0-hadoop2.4.0.jar .Failing this attempt.. Failing the application.
我已经为我的帐户配置了对群集的访问权限.我已经配置了yarn-site.xml.我已经清除了.sparkStaging.我尝试过包含--jars [spark/lib中我的spark程序集的路径].我发现这个问题非常相似,但没有答案.我不知道这是2.1问题,火花1.1.0,kerberized集群,配置,或什么.任何帮助将非常感激.
这可能是因为你留下sparkConf.setMaster("local[n]")
了代码.