Confezione ed esecuzione di Scala Spark Project con Maven

Question

Sto scrivendo un'applicazione in Scala che utilizza Spark. Sto confezionando l'app utilizzando Maven e riscontra problemi durante la costruzione di un "uber" or "fat" jar.

Il problema che sto affrontando è che l'applicazione funziona bene all'interno di un IDE o se fornisco una versione non uber-jar delle dipendenze come il percorso di classe java, ma non funziona il jar uber come percorso classe, ovvero

java -Xmx2G -cp target/spark-example-0.1-SNAPSHOT-jar-with-dependencies.jar debug.spark_example.Example data.txt 

non funziona. Ottengo il seguente messaggio di errore:

ERROR SparkContext: Error initializing SparkContext. 
com.typesafe.config.ConfigException$Missing: No configuration setting found for key 'akka.version' 
    at com.typesafe.config.impl.SimpleConfig.findKey(SimpleConfig.java:124) 
    at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:145) 
    at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:151) 
    at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:159) 
    at com.typesafe.config.impl.SimpleConfig.find(SimpleConfig.java:164) 
    at com.typesafe.config.impl.SimpleConfig.getString(SimpleConfig.java:206) 
    at akka.actor.ActorSystem$Settings.<init>(ActorSystem.scala:168) 
    at akka.actor.ActorSystemImpl.<init>(ActorSystem.scala:504) 
    at akka.actor.ActorSystem$.apply(ActorSystem.scala:141) 
    at akka.actor.ActorSystem$.apply(ActorSystem.scala:118) 
    at org.apache.spark.util.AkkaUtils$.org$apache$spark$util$AkkaUtils$$doCreateActorSystem(AkkaUtils.scala:122) 
    at org.apache.spark.util.AkkaUtils$$anonfun$1.apply(AkkaUtils.scala:54) 
    at org.apache.spark.util.AkkaUtils$$anonfun$1.apply(AkkaUtils.scala:53) 
    at org.apache.spark.util.Utils$$anonfun$startServiceOnPort$1.apply$mcVI$sp(Utils.scala:1991) 
    at scala.collection.immutable.Range.foreach$mVc$sp(Range.scala:160) 
    at org.apache.spark.util.Utils$.startServiceOnPort(Utils.scala:1982) 
    at org.apache.spark.util.AkkaUtils$.createActorSystem(AkkaUtils.scala:56) 
    at org.apache.spark.rpc.akka.AkkaRpcEnvFactory.create(AkkaRpcEnv.scala:245) 
    at org.apache.spark.rpc.RpcEnv$.create(RpcEnv.scala:52) 
    at org.apache.spark.SparkEnv$.create(SparkEnv.scala:247) 
    at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:188) 
    at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:267) 
    at org.apache.spark.SparkContext.<init>(SparkContext.scala:424) 
    at debug.spark_example.Example$.main(Example.scala:9) 
    at debug.spark_example.Example.main(Example.scala) 

Vorrei davvero apprezzare aiuto per capire quello che ho bisogno di aggiungere al file pom.xml e perché ho bisogno di aggiungere per ottenere questo al lavoro.

Ho cercato on-line ed ha trovato le seguenti risorse, che ho provato (vedere nel POM), ma non sono riuscito a mettersi al lavoro:

1) Spark utente Mailing list: http://apache-spark-user-list.1001560.n3.nabble.com/Packaging-a-spark-job-using-maven-td5615.html

2) how to package spark scala application

ho un semplice esempio che illustra il problema, un semplice progetto di classe 1 (src/main/Scala/debug/spark_example/Example.scala):

package debug.spark_example 

import org.apache.spark.{SparkConf, SparkContext} 

object Example { 
    def main(args: Array[String]): Unit = { 
    val sc = new SparkContext(new SparkConf().setAppName("Test").setMaster("local[2]")) 
    val lines = sc.textFile(args(0)) 
    val lineLengths = lines.map(s => s.length) 
    val totalLength = lineLengths.reduce((a, b) => a + b) 
    lineLengths.foreach(println) 
    println(totalLength) 
    } 
} 

Ecco il file pom.xml:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd"> 
    <modelVersion>4.0.0</modelVersion> 
    <groupId>debug.spark-example</groupId> 
    <artifactId>spark-example</artifactId> 
    <version>0.1-SNAPSHOT</version> 
    <inceptionYear>2015</inceptionYear> 
    <properties> 
    <scala.majorVersion>2.11</scala.majorVersion> 
    <scala.minorVersion>.2</scala.minorVersion> 
    <spark.version>1.4.1</spark.version> 
    </properties> 


    <repositories> 
    <repository> 
     <id>scala-tools.org</id> 
     <name>Scala-Tools Maven2 Repository</name> 
     <url>http://scala-tools.org/repo-releases</url> 
    </repository> 
    </repositories> 

    <pluginRepositories> 
    <pluginRepository> 
     <id>scala-tools.org</id> 
     <name>Scala-Tools Maven2 Repository</name> 
     <url>http://scala-tools.org/repo-releases</url> 
    </pluginRepository> 
    </pluginRepositories> 

    <dependencies> 
    <dependency> 
     <groupId>org.scala-lang</groupId> 
     <artifactId>scala-library</artifactId> 
     <version>${scala.majorVersion}${scala.minorVersion}</version> 
    </dependency> 
    <dependency> 
     <groupId>org.apache.spark</groupId> 
     <artifactId>spark-core_${scala.majorVersion}</artifactId> 
     <version>${spark.version}</version> 
    </dependency> 
    </dependencies> 


    <build> 
     <sourceDirectory>src/main/scala</sourceDirectory> 
     <plugins> 
     <plugin> 
      <groupId>org.scala-tools</groupId> 
      <artifactId>maven-scala-plugin</artifactId> 
      <executions> 
      <execution> 
       <goals> 
       <goal>compile</goal> 
       <goal>testCompile</goal> 
       </goals> 
      </execution> 
      </executions> 
     </plugin> 
     <plugin> 
      <groupId>org.apache.maven.plugins</groupId> 
     <artifactId>maven-eclipse-plugin</artifactId> 
     <configuration> 
     <downloadSources>true</downloadSources> 
     <buildcommands> 
      <buildcommand>ch.epfl.lamp.sdt.core.scalabuilder</buildcommand> 
     </buildcommands> 
     <additionalProjectnatures> 
      <projectnature>ch.epfl.lamp.sdt.core.scalanature</projectnature> 
     </additionalProjectnatures> 
     <classpathContainers> 
      <classpathContainer>org.eclipse.jdt.launching.JRE_CONTAINER</classpathContainer> 
      <classpathContainer>ch.epfl.lamp.sdt.launching.SCALA_CONTAINER</classpathContainer> 
     </classpathContainers> 
     </configuration> 
    </plugin> 
    <plugin> 
     <artifactId>maven-assembly-plugin</artifactId> 
     <version>2.4</version> 
     <executions> 
     <execution> 
      <id>make-assembly</id> 
      <phase>package</phase> 
      <goals> 
      <goal>attached</goal> 
      </goals> 
     </execution> 
     </executions> 
     <configuration> 
     <tarLongFileMode>gnu</tarLongFileMode> 
     <descriptorRefs> 
      <descriptorRef>jar-with-dependencies</descriptorRef> 
     </descriptorRefs> 
     </configuration> 
    </plugin> 
    <plugin> 
     <groupId>org.apache.maven.plugins</groupId> 
     <artifactId>maven-shade-plugin</artifactId> 
     <version>2.2</version> 
     <executions> 
     <execution> 
      <phase>package</phase> 
      <goals> 
      <goal>shade</goal> 
      </goals> 
      <configuration> 
      <minimizeJar>false</minimizeJar> 
      <createDependencyReducedPom>false</createDependencyReducedPom> 
      <artifactSet> 
       <includes> 
       <!-- Include here the dependencies you want to be packed in your fat jar --> 
       <include>*:*</include> 
       </includes> 
      </artifactSet> 
      <filters> 
       <filter> 
       <artifact>*:*</artifact> 
       <excludes> 
        <exclude>META-INF/*.SF</exclude> 
        <exclude>META-INF/*.DSA</exclude> 
        <exclude>META-INF/*.RSA</exclude> 
       </excludes> 
       </filter> 
      </filters> 
      <transformers> 
       <transformer implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer"> 
       <resource>reference.conf</resource> 
       </transformer> 
      </transformers> 
      </configuration> 
     </execution> 
     </executions> 
    </plugin> 
    <plugin> 
     <groupId>org.apache.maven.plugins</groupId> 
     <artifactId>maven-surefire-plugin</artifactId> 
     <version>2.7</version> 
     <configuration> 
     <skipTests>true</skipTests> 
     </configuration> 
    </plugin> 
    </plugins> 
</build> 
<reporting> 
    <plugins> 
    <plugin> 
     <groupId>org.scala-tools</groupId> 
     <artifactId>maven-scala-plugin</artifactId> 
    </plugin> 
    </plugins> 
</reporting> 
</project> 

Molte grazie in anticipo per il vostro aiuto.

Answer 1

Questo potrebbe avere qualcosa a che fare con l'ordine dei tuoi plugin Maven. Stai utilizzando i plugin "maven-assembly-plugin" e "maven-shade-plugin" nel tuo progetto, entrambi legati alla stessa fase del ciclo di vita di Maven. Quando ciò accade, Maven esegue i plug-in nell'ordine in cui appaiono nella sezione plugins, quindi nel tuo caso esegue il plug-in di assembly, quindi il plug-in di ombreggiatura.

In base al vaso di uscita che stai cercando di eseguire e alla trasformazione del colore che hai, probabilmente desideri l'ordine opposto. Tuttavia, potrebbe non essere necessario il plug-in di assembly per il tuo caso d'uso. Potresti essere in grado di utilizzare il file target/spark-example-0.1-SNAPSHOT-shaded.jar.

<plugins> 
    <plugin> 
    <groupId>org.apache.maven.plugins</groupId> 
    <artifactId>maven-shade-plugin</artifactId> 
    <!-- SNIP --> 
    </plugin> 
    <plugin> 
    <artifactId>maven-assembly-plugin</artifactId> 
    <!-- SNIP --> 
    </plugin> 
</plugins> 

Answer 2

Pare che la scintilla presentare sceneggiatura deve essere utilizzato per eseguire il programma.

Piuttosto che:

java -Xmx2G -cp target/spark-example-0.1-SNAPSHOT-jar-with-dependencies.jar debug.spark_example.Example data.txt 

fare qualcosa di simile:

<path-to>/spark-1.4.1/bin/spark-submit --class debug.spark_example.Example --master local[2] target/spark-example-0.1-SNAPSHOT-jar-with-dependencies.jar data.txt 

Sembra anche per lavorare, senza il vaso ombreggiato; con solo il jar-with-dependencies. Il seguente file pom.xml ha funzionato per me:

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd"> 
    <modelVersion>4.0.0</modelVersion> 
    <groupId>debug.spark-example</groupId> 
    <artifactId>spark-example</artifactId> 
    <version>0.1-SNAPSHOT</version> 
    <inceptionYear>2015</inceptionYear> 
    <properties> 
     <scala.majorVersion>2.11</scala.majorVersion> 
     <scala.minorVersion>.2</scala.minorVersion> 
     <spark.version>1.4.1</spark.version> 
    </properties> 
    <repositories> 
     <repository> 
      <id>scala-tools.org</id> 
      <name>Scala-Tools Maven2 Repository</name> 
      <url>http://scala-tools.org/repo-releases</url> 
     </repository> 
    </repositories> 
    <pluginRepositories> 
     <pluginRepository> 
      <id>scala-tools.org</id> 
      <name>Scala-Tools Maven2 Repository</name> 
      <url>http://scala-tools.org/repo-releases</url> 
     </pluginRepository> 
    </pluginRepositories> 
    <dependencies> 
     <dependency> 
      <groupId>org.scala-lang</groupId> 
      <artifactId>scala-library</artifactId> 
      <version>${scala.majorVersion}${scala.minorVersion}</version> 
     </dependency> 
     <dependency> 
      <groupId>org.apache.spark</groupId> 
      <artifactId>spark-core_${scala.majorVersion}</artifactId> 
      <version>${spark.version}</version> 
     </dependency> 
    </dependencies> 
    <build> 
     <sourceDirectory>src/main/scala</sourceDirectory> 
     <plugins> 
      <plugin> 
       <groupId>org.scala-tools</groupId> 
       <artifactId>maven-scala-plugin</artifactId> 
       <executions> 
        <execution> 
         <goals> 
          <goal>compile</goal> 
          <goal>testCompile</goal> 
         </goals> 
        </execution> 
       </executions> 
      </plugin> 
      <plugin> 
       <groupId>org.apache.maven.plugins</groupId> 
       <artifactId>maven-eclipse-plugin</artifactId> 
       <configuration> 
        <downloadSources>true</downloadSources> 
        <buildcommands> 
         <buildcommand>ch.epfl.lamp.sdt.core.scalabuilder</buildcommand> 
        </buildcommands> 
        <additionalProjectnatures> 
         <projectnature>ch.epfl.lamp.sdt.core.scalanature</projectnature> 
        </additionalProjectnatures> 
       <classpathContainers> 
        <classpathContainer>org.eclipse.jdt.launching.JRE_CONTAINER</classpathContainer> 
        <classpathContainer>ch.epfl.lamp.sdt.launching.SCALA_CONTAINER</classpathContainer> 
       </classpathContainers> 
       </configuration> 
      </plugin> 
      <plugin> 
       <artifactId>maven-assembly-plugin</artifactId> 
       <version>2.4</version> 
       <executions> 
       <execution> 
        <id>make-assembly</id> 
        <phase>package</phase> 
        <goals> 
         <goal>attached</goal> 
        </goals> 
       </execution> 
      </executions> 
      <configuration> 
       <tarLongFileMode>gnu</tarLongFileMode> 
       <descriptorRefs> 
        <descriptorRef>jar-with-dependencies</descriptorRef> 
       </descriptorRefs> 
      </configuration> 
     </plugin> 
     <plugin> 
      <groupId>org.apache.maven.plugins</groupId> 
      <artifactId>maven-surefire-plugin</artifactId> 
      <version>2.7</version> 
      <configuration> 
       <skipTests>true</skipTests> 
      </configuration> 
     </plugin> 
    </plugins> 
</build> 
<reporting> 
    <plugins> 
     <plugin> 
      <groupId>org.scala-tools</groupId> 
      <artifactId>maven-scala-plugin</artifactId> 
     </plugin> 
    </plugins> 
</reporting> 
</project> 

Answer 3

Akka Docs mi ha aiutato a risolvere il problema.Se stai utilizzando Shade, devi specificare un trasformatore

      <transformer 
            implementation="org.apache.maven.plugins.shade.resource.AppendingTransformer"> 
           <resource>reference.conf</resource> 
          </transformer> 
          <transformer 
            implementation="org.apache.maven.plugins.shade.resource.ManifestResourceTransformer"> 
           <manifestEntries> 
            <Main-Class>akka.Main</Main-Class> 
           </manifestEntries> 
          </transformer>