java.lang.NoClassDefFoundError: kafka/common/TopicAndPartition

Question

Mentre esecuzione del seguente comando nel mio codice:

kafka_streams = [KafkaUtils.createStream(ssc, zk_settings['QUORUM'], zk_settings['CONSUMERS'][k], 
               {zk_settings['TOPICS'][0]: zk_settings['NUM_THREADS']}) 
          .window(zk_settings['WINDOW_DURATION'], zk_settings['SLIDE_DURATION']) 
       for k in range(len(zk_settings['CONSUMERS']))] 

ma sto ottenendo il seguente errore:

Exception in thread "Thread-3" java.lang.NoClassDefFoundError: kafka/common/TopicAndPartition 
at java.lang.Class.getDeclaredMethods0(Native Method) 
at java.lang.Class.privateGetDeclaredMethods(Class.java:2625) 
at java.lang.Class.privateGetPublicMethods(Class.java:2743) 
at java.lang.Class.getMethods(Class.java:1480) 
at py4j.reflection.ReflectionEngine.getMethodsByNameAndLength(ReflectionEngine.java:365) 
at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:317) 
at py4j.reflection.ReflectionEngine.getMethod(ReflectionEngine.java:342) 
at py4j.Gateway.invoke(Gateway.java:252) 
at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133) 
at py4j.commands.CallCommand.execute(CallCommand.java:79) 
at py4j.GatewayConnection.run(GatewayConnection.java:207) 
at java.lang.Thread.run(Thread.java:745) 
Caused by: java.lang.ClassNotFoundException: kafka.common.TopicAndPartition 
at java.net.URLClassLoader$1.run(URLClassLoader.java:366) 
at java.net.URLClassLoader$1.run(URLClassLoader.java:355) 
at java.security.AccessController.doPrivileged(Native Method) 
at java.net.URLClassLoader.findClass(URLClassLoader.java:354) 
at java.lang.ClassLoader.loadClass(ClassLoader.java:425) 
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) 
at java.lang.ClassLoader.loadClass(ClassLoader.java:358) 
... 12 more 

Mi sto perdendo qualcosa?

mi è stato sempre qualche errore di scintilla così ho ricostruito l'errore scintilla e che portano a questo errore.

Answer 1

Si dovrebbe aggiungere --packages quando si invia il codice.

./bin/spark-submit --packages org.apache.spark:spark-streaming-kafka-0-8_2.11:2.2.0 <DIR>/main.py localhost:9092 test 

https://spark.apache.org/docs/latest/streaming-kafka-0-8-integration.html

Answer 2

classNotFoundException significa che quando il programma scintilla presentare è in esecuzione non potrebbe trovare la sua classe richiesta kafka.common.TopicAndPartition nella directory in esecuzione del programma.

un'occhiata ad uso del-presentare scintilla comando :

# spark-submit --help 
Usage: spark-submit [options] <app jar | python file> [app arguments] 
Usage: spark-submit --kill [submission ID] --master [spark://...] 
Usage: spark-submit --status [submission ID] --master [spark://...] 
Usage: spark-submit run-example [options] example-class [example args] 

Options: 
    --master MASTER_URL   spark://host:port, mesos://host:port, yarn, or   local. 
    --deploy-mode DEPLOY_MODE Whether to launch the driver program locally  ("client") or 
           on one of the worker machines inside the cluster ("cluster") 
           (Default: client). 
    --class CLASS_NAME   Your application's main class (for Java/Scala apps). 
    --name NAME     A name of your application. 
    --jars JARS     Comma-separated list of local jars to include on the driver 
           and executor classpaths. 
    --packages     Comma-separated list of maven coordinates of jars to include 
           on the driver and executor classpaths. Will  search the local 
           maven repo, then maven central and any additional remote 
           repositories given by --repositories. The format for the 
           coordinates should be groupId:artifactId:version. 

Aggiungi --jars opzione con il percorso vaso locali di Kafka come segue:

# spark-submit --packages org.apache.spark:spark-streaming-kafka-0-8_2.11:2.2.0 --jars /path/to/org.apache.kafka_kafka_2.11-0.8.2.1.jar,/path/to/com.yammer.metrics_metrics-core-2.2.0.jar your_python_script.py.