Spark-shell a hint, but found not to backspace

Equipped with the Spark cluster, first wrote two small examples with Pyspark, but found that the TAB key is not prompted, so the intention to go to Scala to try, in the Spark-shell under the hint, but found not to backspace, and the hint is not a copy, but the addition, so there is no way to write programs.

Workaround:

1. Open Session Options

2. Terminal-emulation Select Linux in the terminal

3. Map key Check two options

4. This has been successful , but if the remote long-distance operation will interrupt the connection, the next time you need to wait for the operation, in fact, it also affects the use, here also attached solution (optional)

Val Lines =sc.textfile ("Hdfs://alamps:9000/wordcount/input/test.txt")

Lines.count ()

-----
scala> val Lines =sc.textfile ("Hdfs://alamps:9000/wordcount/input/test.txt")
17/10/13 23:09:24 INFO memorystore:ensurefreespace (77922) called with curmem=179665, maxmem=280248975
17/10/13 23:09:24 INFO memorystore:block broadcast_1 stored as values in memory (estimated size 76.1 KB, free 267.0 MB)
17/10/13 23:09:24 INFO memorystore:ensurefreespace (31262) called with curmem=257587, maxmem=280248975
17/10/13 23:09:24 INFO memorystore:block broadcast_1_piece0 stored as bytes in memory (estimated size 30.5 KB, free 267.0 MB)
17/10/13 23:09:24 INFO blockmanagerinfo:added broadcast_1_piece0 in memory on localhost:41619 (size:30.5 KB, free:267.2 MB)
17/10/13 23:09:24 Info blockmanagermaster:updated info of block Broadcast_1_piece0
17/10/13 23:09:24 INFO sparkcontext:created broadcast 1 from Textfile at <console>:12
Lines:org.apache.spark.rdd.rdd[string] = Hdfs://alamps:9000/wordcount/input/test.txt MappedRDD[3] at TextFile at < Console>:12

scala> lines.count ()
17/10/13 23:09:45 INFO fileinputformat:total input paths to process:1
17/10/13 23:09:48 INFO sparkcontext:starting job:count at <console>:15
17/10/13 23:09:48 INFO dagscheduler:got Job 0 (count at <console>:15) with 1 output partitions (allowlocal=false)
17/10/13 23:09:48 INFO dagscheduler:final stage:stage 0 (count at <console>:15)
17/10/13 23:09:48 INFO dagscheduler:parents of Final stage:list ()
17/10/13 23:09:48 INFO dagscheduler:missing parents:list ()
17/10/13 23:09:48 INFO dagscheduler:submitting Stage 0 (hdfs://alamps:9000/wordcount/input/test.txt MappedRDD[3] at Textfile at <console>:12), which have no missing parents
17/10/13 23:09:48 INFO memorystore:ensurefreespace (2544) called with curmem=288849, maxmem=280248975
17/10/13 23:09:48 INFO memorystore:block broadcast_2 stored as values in memory (estimated size 2.5 KB, free 267.0 MB)
17/10/13 23:09:48 INFO Memorystore:ensurefreespace (1898) called with curmem=291393, maxmem=280248975
17/10/13 23:09:48 INFO memorystore:block broadcast_2_piece0 stored as bytes in memory (estimated size 1898.0 B, free 267. 0 MB)
17/10/13 23:09:48 INFO blockmanagerinfo:added broadcast_2_piece0 in memory on localhost:41619 (size:1898.0 B, free:267. 2 MB)
17/10/13 23:09:48 Info blockmanagermaster:updated info of block Broadcast_2_piece0
17/10/13 23:09:48 INFO sparkcontext:created broadcast 2 from broadcast at dagscheduler.scala:838
17/10/13 23:09:48 INFO dagscheduler:submitting 1 missing tasks from Stage 0 (hdfs://alamps:9000/wordcount/input/test.txt MAPPEDRDD[3] at Textfile at <console>:12)
17/10/13 23:09:48 INFO taskschedulerimpl:adding Task set 0.0 with 1 tasks
17/10/13 23:09:48 INFO tasksetmanager:starting task 0.0 in stage 0.0 (TID 0, localhost, any, 1307 bytes)
17/10/13 23:09:48 INFO executor:running task 0.0 in stage 0.0 (TID 0)
17/10/13 23:09:49 INFO hadooprdd:input split:hdfs://alamps:9000/wordcount/input/test.txt:0+88
17/10/13 23:09:49 INFO deprecation:mapred.tip.id is deprecated. Instead, use Mapreduce.task.id
17/10/13 23:09:49 INFO deprecation:mapred.task.id is deprecated. Instead, use Mapreduce.task.attempt.id
17/10/13 23:09:49 INFO deprecation:mapred.task.is.map is deprecated. Instead, use Mapreduce.task.ismap
17/10/13 23:09:49 INFO deprecation:mapred.task.partition is deprecated. Instead, use Mapreduce.task.partition
17/10/13 23:09:49 INFO deprecation:mapred.job.id is deprecated. Instead, use Mapreduce.job.id
17/10/13 23:09:53 INFO executor:finished task 0.0 in stage 0.0 (TID 0). 1920x1080 bytes result sent to driver
17/10/13 23:09:53 INFO dagscheduler:stage 0 (count at <console>:15) finished in 4.875 s
17/10/13 23:09:53 INFO tasksetmanager:finished task 0.0 in stage 0.0 (TID 0) inch 4812 ms on localhost (1/1)
17/10/13 23:09:53 INFO taskschedulerimpl:removed TaskSet 0.0, whose tasks all completed, from pool
17/10/13 23:09:53 INFO dagscheduler:job 0 finished:count at <console>:15, took 5.480197 s
Res2:long = 8



[Email protected] sbin]$ JPS
3596 Master
3733 Worker
2558 DataNode
2748 Secondarynamenode
3814 Jps
2884 ResourceManager
2986 NodeManager
2467 NameNode
[Email protected] sbin]$ Hadoop fs-ls/
Found Items
Drwxr-xr-x-hadoop supergroup 0 2017-10-02 06:29/aaa
Drwxr-xr-x-hadoop supergroup 0 2017-10-06 04:04/external
Drwxr-xr-x-hadoop supergroup 0 2017-10-04 09:14/flowsum
-rw-r--r--1 Hadoop supergroup 2017-10-02 02:52/hello.txt
Drwxr-xr-x-hadoop supergroup 0 2017-10-04 21:10/index
-rw-r--r--1 Hadoop supergroup 143588167 2017-10-01 08:38/jdk-7u65-linux-i586.tar.gz
DRWX-------hadoop supergroup 0 2017-10-05 22:43/tmp
Drwxr-xr-x-hadoop supergroup 0 2017-10-02 06:18/upload
Drwxr-xr-x-hadoop supergroup 0 2017-10-05 22:44/user
Drwxr-xr-x-hadoop supergroup 0 2017-10-03 06:20/WC
Drwxr-xr-x-hadoop supergroup 0 2017-10-01 09:07/wordcount
[Email protected] sbin]$ Hadoop fs-cat/wordcount
Cat: '/wordcount ': is a directory
[Email protected] sbin]$ Hadoop fs-ls/wordcount
Found 2 Items
Drwxr-xr-x-hadoop supergroup 0 2017-10-01 09:00/wordcount/input
Drwxr-xr-x-hadoop supergroup 0 2017-10-01 09:07/wordcount/out
[Email protected] sbin]$ Hadoop fs-ls/wordcount/input
Found 1 Items
-rw-r--r--1 Hadoop supergroup 2017-10-01 09:00/wordcount/input/test.txt
[Email protected] sbin]$ Hadoop fs-cat/wordcount/input/test.txt
Hello, Tom .
Hello Java
Hello C
Hello python
Hello Scala
Hello Spark
Hello baby
Hello Java
[Email protected] sbin]$

Val lines =sc.textfile ("Hdfs://alamps:9000/wordcount/input/test.txt")

Lines.count ()

-----
Scala> Val Lines =sc.textfile ("Hdfs://alamps:9000/wordcount/input/test.txt")
17/10/13 23:09:24 INFO memorystore:ensurefreespace (77922) called with curmem=179665, maxmem=280248975
17/10/13 23:09:24 INFO memorystore:block broadcast_1 stored as values in memory (estimated size 76.1 KB, free 267.0 MB)
17/10/13 23:09:24 INFO memorystore:ensurefreespace (31262) called with curmem=257587, maxmem=280248975
17/10/13 23:09:24 INFO memorystore:block broadcast_1_piece0 stored as bytes in memory (estimated size 30.5 KB, free 267.0 MB)
17/10/13 23:09:24 INFO blockmanagerinfo:added broadcast_1_piece0 in memory on localhost:41619 (size:30.5 KB, free:267.2 MB)
17/10/13 23:09:24 Info blockmanagermaster:updated info of block Broadcast_1_piece0
17/10/13 23:09:24 INFO sparkcontext:created broadcast 1 from Textfile at <console>:12
Lines:org.apache.spark.rdd.rdd[string] = Hdfs://alamps:9000/wordcount/input/test.txt MappedRDD[3] at TextFile at < Console>:12

Scala> Lines.count ()
17/10/13 23:09:45 INFO fileinputformat:total input paths to process:1
17/10/13 23:09:48 INFO sparkcontext:starting job:count at <console>:15
17/10/13 23:09:48 INFO dagscheduler:got Job 0 (count at <console>:15) with 1 output partitions (allowlocal=false)
17/10/13 23:09:48 INFO dagscheduler:final stage:stage 0 (count at <console>:15)
17/10/13 23:09:48 INFO dagscheduler:parents of Final stage:list ()
17/10/13 23:09:48 INFO dagscheduler:missing parents:list ()
17/10/13 23:09:48 INFO dagscheduler:submitting Stage 0 (hdfs://alamps:9000/wordcount/input/test.txt MappedRDD[3] at Textfile at <console>:12), which have no missing parents
17/10/13 23:09:48 INFO memorystore:ensurefreespace (2544) called with curmem=288849, maxmem=280248975
17/10/13 23:09:48 INFO memorystore:block broadcast_2 stored as values in memory (estimated size 2.5 KB, free 267.0 MB)
17/10/13 23:09:48 INFO Memorystore:ensurefreespace (1898) called with curmem=291393, maxmem=280248975
17/10/13 23:09:48 INFO memorystore:block broadcast_2_piece0 stored as bytes in memory (estimated size 1898.0 B, free 267. 0 MB)
17/10/13 23:09:48 INFO blockmanagerinfo:added broadcast_2_piece0 in memory on localhost:41619 (size:1898.0 B, free:267. 2 MB)
17/10/13 23:09:48 Info blockmanagermaster:updated info of block Broadcast_2_piece0
17/10/13 23:09:48 INFO sparkcontext:created broadcast 2 from broadcast at dagscheduler.scala:838
17/10/13 23:09:48 INFO dagscheduler:submitting 1 missing tasks from Stage 0 (hdfs://alamps:9000/wordcount/input/test.txt MAPPEDRDD[3] at Textfile at <console>:12)
17/10/13 23:09:48 INFO taskschedulerimpl:adding Task set 0.0 with 1 tasks
17/10/13 23:09:48 INFO tasksetmanager:starting task 0.0 in stage 0.0 (TID 0, localhost, any, 1307 bytes)
17/10/13 23:09:48 INFO executor:running task 0.0 in stage 0.0 (TID 0)
17/10/13 23:09:49 INFO hadooprdd:input split:hdfs://alamps:9000/wordcount/input/test.txt:0+88
17/10/13 23:09:49 INFO deprecation:mapred.tip.id is deprecated. Instead, use Mapreduce.task.id
17/10/13 23:09:49 INFO deprecation:mapred.task.id is deprecated. Instead, use Mapreduce.task.attempt.id
17/10/13 23:09:49 INFO deprecation:mapred.task.is.map is deprecated. Instead, use Mapreduce.task.ismap
17/10/13 23:09:49 INFO deprecation:mapred.task.partition is deprecated. Instead, use Mapreduce.task.partition
17/10/13 23:09:49 INFO deprecation:mapred.job.id is deprecated. Instead, use Mapreduce.job.id
17/10/13 23:09:53 INFO executor:finished task 0.0 in stage 0.0 (TID 0). 1920x1080 bytes result sent to driver
17/10/13 23:09:53 INFO dagscheduler:stage 0 (count at <console>:15) finished in 4.875 s
17/10/13 23:09:53 INFO tasksetmanager:finished task 0.0 in stage 0.0 (TID 0) inch 4812 ms on localhost (1/1)
17/10/13 23:09:53 INFO taskschedulerimpl:removed TaskSet 0.0, whose tasks all completed, from pool
17/10/13 23:09:53 INFO dagscheduler:job 0 finished:count at <console>:15, took 5.480197 s
Res2:long = 8



[Email protected] sbin]$ JPS
3596 Master
3733 Worker
2558 DataNode
2748 Secondarynamenode
3814 Jps
2884 ResourceManager
2986 NodeManager
2467 NameNode
[Email protected] sbin]$ Hadoop fs-ls/
Found Items
Drwxr-xr-x-hadoop supergroup 0 2017-10-02 06:29/aaa
Drwxr-xr-x-hadoop supergroup 0 2017-10-06 04:04/external
Drwxr-xr-x-hadoop supergroup 0 2017-10-04 09:14/flowsum
-rw-r--r--1 Hadoop supergroup 2017-10-02 02:52/hello.txt
Drwxr-xr-x-hadoop supergroup 0 2017-10-04 21:10/index
-rw-r--r--1 Hadoop supergroup 143588167 2017-10-01 08:38/jdk-7u65-linux-i586.tar.gz
DRWX-------hadoop supergroup 0 2017-10-05 22:43/tmp
Drwxr-xr-x-hadoop supergroup 0 2017-10-02 06:18/upload
Drwxr-xr-x-hadoop supergroup 0 2017-10-05 22:44/user
Drwxr-xr-x-hadoop supergroup 0 2017-10-03 06:20/WC
Drwxr-xr-x-hadoop supergroup 0 2017-10-01 09:07/wordcount
[Email protected] sbin]$ Hadoop fs-cat/wordcount
Cat: '/wordcount ': is a directory
[Email protected] sbin]$ Hadoop fs-ls/wordcount
Found 2 Items
Drwxr-xr-x-hadoop supergroup 0 2017-10-01 09:00/wordcount/input
Drwxr-xr-x-hadoop supergroup 0 2017-10-01 09:07/wordcount/out
[Email protected] sbin]$ Hadoop fs-ls/wordcount/input
Found 1 Items
-rw-r--r--1 Hadoop supergroup 2017-10-01 09:00/wordcount/input/test.txt
[Email protected] sbin]$ Hadoop fs-cat/wordcount/input/test.txt
Hello, Tom.
Hello Java
Hello C
Hello python
Hello Scala
Hello Spark
Hello Baby
Hello Java
[Email protected] sbin]$

Spark-shell a hint, but found not to backspace