Linee di comando shell Spark

Question

Sono nuovo di Spark e sto cercando di capire come posso usare la shell Spark.

Guardato nella documentazione del sito di Spark e non mostra come creare directory o come vedere tutti i miei file in spark shell. Se qualcuno potesse aiutarmi, lo apprezzerei.

Answer 1

In questo contesto si può presumere che Spark shell sia solo un normale Scala REPL quindi si applicano le stesse regole. È possibile ottenere un elenco dei comandi disponibili utilizzando :help.

Welcome to 
     ____    __ 
    /__/__ ___ _____/ /__ 
    _\ \/ _ \/ _ `/ __/ '_/ 
    /___/ .__/\_,_/_/ /_/\_\ version 2.3.0 
     /_/ 

Using Scala version 2.11.8 (OpenJDK 64-Bit Server VM, Java 1.8.0_151) 
Type in expressions to have them evaluated. 
Type :help for more information. 

scala> :help 
All commands can be abbreviated, e.g., :he instead of :help. 
:edit <id>|<line>  edit history 
:help [command]   print this summary or command-specific help 
:history [num]   show the history (optional num is commands to show) 
:h? <string>    search the history 
:imports [name name ...] show import history, identifying sources of names 
:implicits [-v]   show the implicits in scope 
:javap <path|class>  disassemble a file or class name 
:line <id>|<line>  place line(s) at the end of history 
:load <path>    interpret lines in a file 
:paste [-raw] [path]  enter paste mode or paste a file 
:power     enable power user mode 
:quit     exit the interpreter 
:replay [options]  reset the repl and replay all previous commands 
:require <path>   add a jar to the classpath 
:reset [options]   reset the repl to its initial state, forgetting all session entries 
:save <path>    save replayable session to a file 
:sh <command line>  run a shell command (result is implicitly => List[String]) 
:settings <options>  update compiler options, if possible; see reset 
:silent     disable/enable automatic printing of results 
:type [-v] <expr>  display the type of an expression without evaluating it 
:kind [-v] <expr>  display the kind of expression's type 
:warnings    show the suppressed warnings from the most recent line which had any 

Come potete vedere qui sopra è possibile richiamare i comandi della shell utilizzando :sh. Ad esempio:

scala> :sh mkdir foobar 
res0: scala.tools.nsc.interpreter.ProcessResult = `mkdir foobar` (0 lines, exit 0) 

scala> :sh touch foobar/foo 
res1: scala.tools.nsc.interpreter.ProcessResult = `touch foobar/foo` (0 lines, exit 0) 

scala> :sh touch foobar/bar 
res2: scala.tools.nsc.interpreter.ProcessResult = `touch foobar/bar` (0 lines, exit 0) 

scala> :sh ls foobar 
res3: scala.tools.nsc.interpreter.ProcessResult = `ls foobar` (2 lines, exit 0) 

scala> res3.line foreach println 
line lines 

scala> res3.lines foreach println 
bar 
foo 

Answer 2

Click for image description here

: q o: comando quit viene utilizzato per uscire dalla scala ripple.