Spark: Eccezione nel thread "main" java.lang.VerifyError: class com.fasterxml.jackson.module.scala.ser.ScalaIteratorSerializer

Question

Sto provando a eseguire parallelamente Spark in alcuni dati all'interno di un grande progetto, ma anche su l'esempio più semplice mi viene visualizzato questo errore

Exception in thread "main" java.lang.VerifyError: class com.fasterxml.jackson.module.scala.ser.ScalaIteratorSerializer overrides final method withResolved.(Lcom/fasterxml/jackson/databind/BeanProperty;Lcom/fasterxml/jackson/databind/jsontype/TypeSerializer;Lcom/fasterxml/jackson/databind/JsonSerializer;)Lcom/fasterxml/jackson/databind/ser/std/AsArraySerializerBase; 

L'errore si presenta con un semplice parallelismo, anche questo semplice. Non ho idea di dove questo errore è venuto anche da

val conf: SparkConf = new SparkConf().setAppName("IEEG Spark").setMaster("local") 
    val sc: SparkContext = new SparkContext(conf) 
    val data = Array(1, 2, 3, 4, 5) 
    val distVals = sc.parallelize(data) 
    distVals.foreach(println) 

e il seguente è il mio esperto di file di pom.xml

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
     xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd"> 

    <groupId>astiefel</groupId> 
    <artifactId>ieeg-spark</artifactId> 
    <modelVersion>4.0.0</modelVersion> 
    <name>Spark IEEG</name> 
    <parent> 
     <groupId>edu.upenn.cis.ieeg</groupId> 
     <artifactId>ieeg</artifactId> 
     <version>1.15-SNAPSHOT</version> 
    </parent> 
    <properties> 
     <scala.version>2.10.4</scala.version> 
    </properties> 
    <dependencies> 
     <dependency> 
      <groupId>edu.upenn.cis.ieeg</groupId> 
      <artifactId>ieeg-client</artifactId> 
     </dependency> 
     <dependency> 
      <groupId>org.apache.spark</groupId> 
      <artifactId>spark-core_2.10</artifactId> 
      <version>1.5.0</version> 
     </dependency> 
     <dependency> 
      <groupId>org.scala-lang</groupId> 
      <artifactId>scala-compiler</artifactId> 
      <version>${scala.version}</version> 
      <scope>compile</scope> 
     </dependency> 
     <dependency> 
      <groupId>org.scalanlp</groupId> 
      <artifactId>breeze_2.10</artifactId> 
      <version>0.10</version> 
     </dependency> 
    </dependencies> 

    <build> 
     <sourceDirectory>src/main/scala</sourceDirectory> 
     <plugins> 
      <plugin> 
       <groupId>org.scala-tools</groupId> 
       <artifactId>maven-scala-plugin</artifactId> 
       <executions> 
        <execution> 
         <goals> 
          <goal>compile</goal> 
          <goal>testCompile</goal> 
         </goals> 
        </execution> 
       </executions> 
       <configuration> 
        <scalaVersion>${scala.version}</scalaVersion> 
        <args> 
         <arg>-target:jvm-1.5</arg> 
        </args> 
       </configuration> 
      </plugin> 
      <plugin> 
       <groupId>org.apache.maven.plugins</groupId> 
       <artifactId>maven-eclipse-plugin</artifactId> 
       <configuration> 
        <downloadSources>true</downloadSources> 
        <buildcommands> 
         <buildcommand>ch.epfl.lamp.sdt.core.scalabuilder</buildcommand> 
        </buildcommands> 
        <additionalProjectnatures> 
         <projectnature>ch.epfl.lamp.sdt.core.scalanature</projectnature> 
        </additionalProjectnatures> 
        <classpathContainers> 
         <classpathContainer>org.eclipse.jdt.launching.JRE_CONTAINER</classpathContainer> 
         <classpathContainer>ch.epfl.lamp.sdt.launching.SCALA_CONTAINER</classpathContainer> 
        </classpathContainers> 
       </configuration> 
      </plugin> 
     </plugins> 
    </build> 
    <reporting> 
     <plugins> 
      <plugin> 
       <groupId>org.scala-tools</groupId> 
       <artifactId>maven-scala-plugin</artifactId> 
       <configuration> 
        <scalaVersion>${scala.version}</scalaVersion> 
       </configuration> 
      </plugin> 
     </plugins> 
    </reporting> 
</project> 

Sono così perso con questo errore, qualche consiglio per quanto riguarda da dove cominciare?

Answer 1

Si potrebbe eseguire un semplice test di parallelizzazione Spark con dipendenza inferiore, utilizzare spark-assembly-xxx.jar direttamente come dipendenza jar.

<profile> 
     <id>apache-spark/scala</id> 
     <dependencies> 
      <!--Apache Spark --> 
      <dependency> 
       <groupId>org.apache.spark</groupId> 
       <artifactId>spark-assembly</artifactId> 
       <version>1.5.2</version> 
       <scope>system</scope> 
       <systemPath>${spark.home}/lib/spark-assembly-1.5.2-hadoop2.6.0.jar</systemPath> 
      </dependency> 
      <dependency> 
       <groupId>org.scala-lang</groupId> 
       <artifactId>scala-library</artifactId> 
       <version>2.10.2</version> 
      </dependency> 
     </dependencies> 
    </profile>