我使用的是spark 0.91
与MLlib 0.91
上DSE
尝试在独立模式下运行以下代码时
val parsedData = sc.parallelize((1 to 1000). map { line => LabeledPoint(0.0, Array(0.0, 0.4, 0.3)) }) val numIterations = 2 val model = LinearRegressionWithSGD.train(parsedData, numIterations)
我收到这个错误:
14/09/20 14:28:37 ERROR OneForOneStrategy: org.jblas.DoubleMatrix cannot be cast to org.jblas.DoubleMatrix java.lang.ClassCastException: org.jblas.DoubleMatrix cannot be cast to org.jblas.DoubleMatrix at org.apache.spark.mllib.optimization.GradientDescent$$anonfun$runMiniBatchSGD$1$$anonfun$2.apply(GradientDescent.scala:150) at org.apache.spark.mllib.optimization.GradientDescent$$anonfun$runMiniBatchSGD$1$$anonfun$2.apply(GradientDescent.scala:150) at org.apache.spark.rdd.RDD$$anonfun$6.apply(RDD.scala:677) at org.apache.spark.rdd.RDD$$anonfun$6.apply(RDD.scala:674) at org.apache.spark.scheduler.JobWaiter.taskSucceeded(JobWaiter.scala:56) at org.apache.spark.scheduler.DAGScheduler.handleTaskCompletion(DAGScheduler.scala:846) at org.apache.spark.scheduler.DAGScheduler.processEvent(DAGScheduler.scala:601)
只有在尝试运行独立应用程序时才会发生这种情况.它适用于火花壳(dse spark).有任何想法吗?
更新:
当我在REPL上创建一个对象时,getClassLoader返回:
scala> new org.jblas.DoubleMatrix().getClass().getClassLoader() res3: ClassLoader = ModuleClassLoader:Analytics
但是当我在独立模式下运行(使用spark类)时,它会返回
new org.jblas.DoubleMatrix().getClass().getClassLoader(): class= SystemClassLoader
也许这是一个提示.
我使用SBT生成jar并使用spark-class提交它.这是配置
name := "analytics" version := "1.0" scalaVersion := "2.10.3" unmanagedJars in Compile ++= Attributed.blankSeq((file("./dse/lib/") * "*.jar").get) unmanagedJars in Compile ++= Attributed.blankSeq((file("./dse/resources/spark/lib/") * "*.jar").get) unmanagedJars in Compile ++= Attributed.blankSeq((file("./dse/resources/cassandra/lib/") * "*.jar").get) unmanagedJars in Runtime ++= Attributed.blankSeq((file("./dse/resources/hadoop/") * "*.jar").get) unmanagedJars in Runtime ++= Attributed.blankSeq((file("./dse/resources/hadoop/lib/") * "*.jar").get) unmanagedJars in Compile ++= Attributed.blankSeq((file("./dse/resources/driver/lib/") * "*.jar").get)
更新2:使用dse演示的配置来构建和部署ant但我再次面临同样的错误