sparkstreaming Introduction and examples

Reading probably understand the principle of the next streaming, but the wood has moved hands ah ... The beginning of everything is difficult, a wordcount 2 hours how can not run out results. I'm so stupid, okay.

Sparkstreaming is a batch-based streaming computing framework for scenarios where real-time data is mixed with historical data (for example, you read real-time data into processing with streaming, and then use Sparksql to extract historical data and associate it with it). Spark streaming splits the data stream into an rdd in time slices and uses an RDD operation to process each piece of data, with no block of data generating a spark job for processing, and finally processing the data for each time slice in batch mode. (many will not explain, Baidu is good ~)

First make sure that you have Hadoop and spark installed and that the jar package is already in the idea.

Write It--

Novice to pay attention to the Red box section, the Spark official online example is to call the Socketfilestream method, which is remote through the socket connection, if only on the local test learning, using Textfilestream to read the native file path, yes, the path is not a file, Because sparkstreaming is processing real-time data, if you specify a file directly, the output is not able to get results. So a new path is created, where seconds (20) is set to read every 20 seconds. Run a bit later.

After booting, the prepared file CP is set to this path, and after 20 seconds The result comes out, simulating the next real-time data. End.

sparkstreaming Introduction and examples