Mac scintilla scocca Errore durante l'inizializzazione SparkContext

Question

ho cercato di iniziare scintilla 1.6.0 (spark-1.6.0-bin-hadoop2.4) su Mac OS Yosemite 10.10.5 utilizzando

"./bin/spark-shell". 

Ha l'errore sotto. Ho anche provato ad installare diverse versioni di Spark ma tutte hanno lo stesso errore. Questa è la seconda volta che eseguo Spark. La mia precedente esecuzione funziona bene.

log4j:WARN No appenders could be found for logger (org.apache.hadoop.metrics2.lib.MutableMetricsFactory). 
log4j:WARN Please initialize the log4j system properly. 
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info. 
Using Spark's repl log4j profile: org/apache/spark/log4j-defaults-repl.properties 
To adjust logging level use sc.setLogLevel("INFO") 
Welcome to 
     ____    __ 
    /__/__ ___ _____/ /__ 
    _\ \/ _ \/ _ `/ __/ '_/ 
    /___/ .__/\_,_/_/ /_/\_\ version 1.6.0 
     /_/ 

Using Scala version 2.10.5 (Java HotSpot(TM) 64-Bit Server VM, Java 1.7.0_79) 
Type in expressions to have them evaluated. 
Type :help for more information. 
16/01/04 13:49:40 WARN Utils: Service 'sparkDriver' could not bind on port 0. Attempting port 1. 
16/01/04 13:49:40 WARN Utils: Service 'sparkDriver' could not bind on port 0. Attempting port 1. 
16/01/04 13:49:40 WARN Utils: Service 'sparkDriver' could not bind on port 0. Attempting port 1. 
16/01/04 13:49:40 WARN Utils: Service 'sparkDriver' could not bind on port 0. Attempting port 1. 
16/01/04 13:49:40 WARN Utils: Service 'sparkDriver' could not bind on port 0. Attempting port 1. 
16/01/04 13:49:40 WARN Utils: Service 'sparkDriver' could not bind on port 0. Attempting port 1. 
16/01/04 13:49:40 WARN Utils: Service 'sparkDriver' could not bind on port 0. Attempting port 1. 
16/01/04 13:49:40 WARN Utils: Service 'sparkDriver' could not bind on port 0. Attempting port 1. 
16/01/04 13:49:40 WARN Utils: Service 'sparkDriver' could not bind on port 0. Attempting port 1. 
16/01/04 13:49:40 WARN Utils: Service 'sparkDriver' could not bind on port 0. Attempting port 1. 
16/01/04 13:49:40 WARN Utils: Service 'sparkDriver' could not bind on port 0. Attempting port 1. 
16/01/04 13:49:40 WARN Utils: Service 'sparkDriver' could not bind on port 0. Attempting port 1. 
16/01/04 13:49:40 WARN Utils: Service 'sparkDriver' could not bind on port 0. Attempting port 1. 
16/01/04 13:49:40 WARN Utils: Service 'sparkDriver' could not bind on port 0. Attempting port 1. 
16/01/04 13:49:40 WARN Utils: Service 'sparkDriver' could not bind on port 0. Attempting port 1. 
16/01/04 13:49:40 WARN Utils: Service 'sparkDriver' could not bind on port 0. Attempting port 1. 
16/01/04 13:49:40 ERROR SparkContext: Error initializing SparkContext. 
java.net.BindException: Can't assign requested address: Service 'sparkDriver' failed after 16 retries! 
    at sun.nio.ch.Net.bind0(Native Method) 
    at sun.nio.ch.Net.bind(Net.java:444) 
    at sun.nio.ch.Net.bind(Net.java:436) 
    at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214) 
    at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) 
    at io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:125) 
    at io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:485) 
    at io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1089) 
    at io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:430) 
    at io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:415) 
    at io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:903) 
    at io.netty.channel.AbstractChannel.bind(AbstractChannel.java:198) 
    at io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:348) 
    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357) 
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357) 
    at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) 
    at java.lang.Thread.run(Thread.java:745) 
java.net.BindException: Can't assign requested address: Service 'sparkDriver' failed after 16 retries! 
    at sun.nio.ch.Net.bind0(Native Method) 
    at sun.nio.ch.Net.bind(Net.java:444) 
    at sun.nio.ch.Net.bind(Net.java:436) 
    at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214) 
    at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) 
    at io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:125) 
    at io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:485) 
    at io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1089) 
    at io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:430) 
    at io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:415) 
    at io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:903) 
    at io.netty.channel.AbstractChannel.bind(AbstractChannel.java:198) 
    at io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:348) 
    at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:357) 
    at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:357) 
    at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:111) 
    at java.lang.Thread.run(Thread.java:745) 

java.lang.NullPointerException 
    at org.apache.spark.sql.SQLContext$.createListenerAndUI(SQLContext.scala:1367) 
    at org.apache.spark.sql.hive.HiveContext.<init>(HiveContext.scala:101) 
    at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) 
    at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57) 
    at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) 
    at java.lang.reflect.Constructor.newInstance(Constructor.java:526) 
    at org.apache.spark.repl.SparkILoop.createSQLContext(SparkILoop.scala:1028) 
    at $iwC$$iwC.<init>(<console>:15) 
    at $iwC.<init>(<console>:24) 
    at <init>(<console>:26) 
    at .<init>(<console>:30) 
    at .<clinit>(<console>) 
    at .<init>(<console>:7) 
    at .<clinit>(<console>) 
    at $print(<console>) 
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) 
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
    at java.lang.reflect.Method.invoke(Method.java:606) 
    at org.apache.spark.repl.SparkIMain$ReadEvalPrint.call(SparkIMain.scala:1065) 
    at org.apache.spark.repl.SparkIMain$Request.loadAndRun(SparkIMain.scala:1346) 
    at org.apache.spark.repl.SparkIMain.loadAndRunReq$1(SparkIMain.scala:840) 
    at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871) 
    at org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:819) 
    at org.apache.spark.repl.SparkILoop.reallyInterpret$1(SparkILoop.scala:857) 
    at org.apache.spark.repl.SparkILoop.interpretStartingWith(SparkILoop.scala:902) 
    at org.apache.spark.repl.SparkILoop.command(SparkILoop.scala:814) 
    at org.apache.spark.repl.SparkILoopInit$$anonfun$initializeSpark$1.apply(SparkILoopInit.scala:132) 
    at org.apache.spark.repl.SparkILoopInit$$anonfun$initializeSpark$1.apply(SparkILoopInit.scala:124) 
    at org.apache.spark.repl.SparkIMain.beQuietDuring(SparkIMain.scala:324) 
    at org.apache.spark.repl.SparkILoopInit$class.initializeSpark(SparkILoopInit.scala:124) 
    at org.apache.spark.repl.SparkILoop.initializeSpark(SparkILoop.scala:64) 
    at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1$$anonfun$apply$mcZ$sp$5.apply$mcV$sp(SparkILoop.scala:974) 
    at org.apache.spark.repl.SparkILoopInit$class.runThunks(SparkILoopInit.scala:159) 
    at org.apache.spark.repl.SparkILoop.runThunks(SparkILoop.scala:64) 
    at org.apache.spark.repl.SparkILoopInit$class.postInitialization(SparkILoopInit.scala:108) 
    at org.apache.spark.repl.SparkILoop.postInitialization(SparkILoop.scala:64) 
    at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply$mcZ$sp(SparkILoop.scala:991) 
    at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945) 
    at org.apache.spark.repl.SparkILoop$$anonfun$org$apache$spark$repl$SparkILoop$$process$1.apply(SparkILoop.scala:945) 
    at scala.tools.nsc.util.ScalaClassLoader$.savingContextLoader(ScalaClassLoader.scala:135) 
    at org.apache.spark.repl.SparkILoop.org$apache$spark$repl$SparkILoop$$process(SparkILoop.scala:945) 
    at org.apache.spark.repl.SparkILoop.process(SparkILoop.scala:1059) 
    at org.apache.spark.repl.Main$.main(Main.scala:31) 
    at org.apache.spark.repl.Main.main(Main.scala) 
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) 
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) 
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) 
    at java.lang.reflect.Method.invoke(Method.java:606) 
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:731) 
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:181) 
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:206) 
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:121) 
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala) 

<console>:16: error: not found: value sqlContext 
     import sqlContext.implicits._ 
       ^
<console>:16: error: not found: value sqlContext 
     import sqlContext.sql 

poi aggiungo

export SPARK_LOCAL_IP="127.0.0.1" 

a spark-env.sh, cambiamenti di errore a:

ERROR : No route to host 
    java.net.ConnectException: No route to host 
     at java.net.Inet6AddressImpl.isReachable0(Native Method) 
     at java.net.Inet6AddressImpl.isReachable(Inet6AddressImpl.java:77) 
     at java.net.InetAddress.isReachable(InetAddress.java:475) 
... 
<console>:10: error: not found: value sqlContext 
     import sqlContext.implicits._ 
      ^
<console>:10: error: not found: value sqlContext 
     import sqlContext.sql 

Answer 1

ho costruito dal branch master corrente con la versione 2.0.0-SNAPSHOT. Dopo aver aggiunto export SPARK_LOCAL_IP="127.0.0.1" a load-spark-env.sh ha funzionato per me. Sto usando Macos 10.10.5. Quindi potrebbe essere un problema di versione?

Answer 2

passi Seguendo potrebbero aiutare:

1. Crea il tuo nome host utilizzando il comando "hostname".

2. Fai una voce nel file/etc/hosts per il nome host se non è presente come segue:

   127.0.0.1  your_hostname 
   

Spero che questo aiuti !!

Answer 3

A volte il firewall impedisce la creazione e il binding di un socket. assicurarsi che il firewall non è abilitare e inoltre si deve controllare l'IP della macchina in/etc/hosts e assicurarsi che sia OK quindi riprovare:

sudo ufw disable 

Answer 4

Ho sempre trovato che quando si passa tra le reti. Questo risolve:

$ sudo hostname -s 127.0.0.1

Answer 5

Basta impostare il spark.driver.host per essere il vostro localhost se si utilizza IDE

SparkConf conf = new SparkConf().setMaster("local[2]").setAppName("AnyName").set("spark.driver.host", "localhost"); 
JavaSparkContext sc = new JavaSparkContext(conf); 

Answer 6

Se non si desidera cambiare il nome host del Mac, è possibile effettuare le seguente:

1. Trova il file modello spark-env.sh.template sulla vostra macchina (E 'probabilmente in /usr/local/Cellar/apache-spark/2.1.0/libexec/conf/).
2. cp spark-env.sh.template spark-env.sh
3. Aggiungi export SPARK_LOCAL_IP=127.0.0.1 sotto il commento per IP locale.

Inizio spark-shell e divertiti.

Answer 7

Ci sono due errori che penso.

1. L'IP locale spark non era corretto e deve essere modificato in 127.0.0.1.
2. Non hai definito correttamente sqlContext.

Per 1. Ho provato:

• 1) esportati SPARK_LOCAL_IP = "127.0.0.1" in ~/.bash_profile
• 2) aggiunto export SPARK_LOCAL_IP = "127.0.0.1" in del carico spark-env.sh sotto $ SPARK_HOME

Ma nessuno dei due ha funzionato. Poi ho provato quanto segue e ha funzionato:

val conf = new SparkConf(). 
    setAppName("SparkExample"). 
    setMaster("local[*]"). 
    set("spark.driver.bindAddress","127.0.0.1") 
val sc = new SparkContext(conf) 

Per 2. si può provare:

sqlContext = SparkSession.builder.config("spark.master","local[*]").getOrCreate() 

e poi import sqlContext.implicits._

Il generatore di SparkSession utilizzerà automaticamente lo SparkContext se esiste, altrimenti ne creerà uno. È possibile crearne due esplicitamente, se necessario.