Spark API Programming Hands-on -05-spark file operation and debug

This time we start Spark-shell by specifying the Executor-memory parameter:

The boot was successful.

On the command line we have specified that the memory of executor on each machine Spark-shell run take up is 1g in size, and after successful launch see Web page:

To read files from HDFs:

The Mappedrdd returned in the command line, using todebugstring, can view its lineage relationship:

You can see that Mappedrdd was converted from Hadooprdd.

Then look at the source code of Textfile:

Hadoopfile This method returns a Hadooprdd, the source code is as follows:

And the map method produces a mappedrdd:

Here's a simple wordcount operation:

Execution Result:

Use todebugstring again to view the following dependencies:

Hadooprdd, Mappedrdd, Flatmappedrdd, Mappedrdd, Shuffledrdd

Spark API Programming Hands-on -05-spark file operation and debug