Apache Beam WordCount Programming Combat and Source code interpretation

Overview: Apache Beam WordCount Programming Combat and source interpretation, and through IntelliJ idea and terminal two ways to debug and run WordCount program, Apache Beam batch processing and streaming of big data, Provides a set of advanced unified programming models and can run on large data processing engines. Full project GitHub source

Responsible for the company Big Data processing related architecture, but has the diversity, greatly increased the development cost, the urgent need unified programming processing, Apache Beam, one place programming, everywhere runs, therefore will toss the result to share out.

1.Apache Beam Programming-preface, Apache beam features and key concepts.

Apache Beam became Apache's new top-level project on January 10, 2017.

1.1.Apache Beam Features:

• Unified: Uses a single programming model for batch and streaming media use cases.
• Convenient: Supports multiple pipelines environments, including: Apache Apex, Apache Flink, Apache Spark, and Google Cloud Dataflow.
• Extensible: Write and share new Sdks,io connectors and transformation libraries
  Partial translation from official website: Apacher Beam official website

1.2.Apache Beam Key concept: 1.2.1.Apache Beam SDKs

The main development API is to provide a unified programming model for batch processing and stream processing. Currently (2017) supports the Java language, while Python is under intense development.

1.2.2 Apache Beam Pipeline Runners (Beam actuators/performers), support Apache Apex,apache flink,apache spark,google Cloud Dataflow multiple Big Data computing frameworks. It is an Apache beam programming, multi-compute framework running. 1.2.3. Their support for the following details

2.Apache beam Programming Combat –apache Beam Source code interpretation

View the full project GitHub source based on Maven,intellij IDEA,POM.XM. Import the full project directly via the idea's project import feature, wait for maven to download the dependency package, and then follow the interpretation steps below to run smoothly.

2.1. Source parsing-apache Beam Data Flow processing principle Analysis:

Key steps:

• Create pipeline
• Apply a transform to pipeline
• Read input file
• Apply Pardo Conversion
• Apply the conversion provided by the SDK (for example: Count)
• Write out the output
• Run pipeline

2.2. Source code parsing, complete project GitHub source code, attached wordcount,pom.xml, etc.

/** * MIT * Author:wangxiaolei (Wang Xiaole). * date:17-2-20. * Project:apachebeamwordcount. *ImportOrg.apache.beam.sdk.Pipeline;ImportOrg.apache.beam.sdk.io.TextIO;ImportOrg.apache.beam.sdk.options.Default;ImportOrg.apache.beam.sdk.options.Description;ImportOrg.apache.beam.sdk.options.PipelineOptions;ImportOrg.apache.beam.sdk.options.PipelineOptionsFactory;Importorg.apache.beam.sdk.options.Validation.Required;ImportOrg.apache.beam.sdk.transforms.Aggregator;ImportOrg.apache.beam.sdk.transforms.Count;ImportOrg.apache.beam.sdk.transforms.DoFn;Importorg.apache.beam.sdk.transforms.MapElements;ImportOrg.apache.beam.sdk.transforms.PTransform;ImportOrg.apache.beam.sdk.transforms.ParDo;ImportOrg.apache.beam.sdk.transforms.SimpleFunction;ImportOrg.apache.beam.sdk.transforms.Sum;ImportOrg.apache.beam.sdk.values.KV;ImportOrg.apache.beam.sdk.values.pcollection;public class WordCount {    /** *1.a. Dofn Programming Pipeline makes the code very concise.     B. The input text to do Word division, output. */Static class extractwordsfn extends dofn<string, string> {         Private FinalAggregator<long, long> emptylines = Createaggregator ("Emptylines", Sum.oflongs ());@ProcessElementpublic void Processelement (Processcontext c) {if(C.element (). Trim (). IsEmpty ()) {Emptylines.addvalue (1L); }//divide lines of text into wordsstring[] Words = C.element (). Split ("[^a-za-z ']+");//Output the word in pcollection             for(String word:words) {if(!word.isempty ())                {c.output (word); }            }        }    }/**. Formats the input text data, converting the word to and counting the printed string. */public static class formatastextfn extends simplefunction<KV<String , Long;, String> {        @OverridePublic String Apply (kv<string, long> input) {returnInput.getkey () +": "+ Input.getvalue (); }    }/**. Word count, Ptransform (pcollection Transform) converts the pcollection text line to a formatted, counted word. */public static class countwords extends ptransform<pcollection<String ,Pcollection<kv<string, long>>> {@OverridePublic pcollection<kv<string, long>> expand (pcollection<string> lines) {//Convert lines of text to a single wordpcollection<string> words = lines.apply (Pardo.of (NewEXTRACTWORDSFN ()));//Calculate the number of words per wordpcollection<kv<string, long>> wordcounts = words.apply (Count.<string>perelement ());returnwordcounts; }    }/**. You can customize some options, such as the file input/output path */public interface Wordcountoptionsextendspipelineoptions {/** * File input option, you can pass in the path parameter through the command line, the path defaults to Gs://apache-beam-samples/shakespeare/kinglear.txt * /        @Description("Path of the file to read from")@Default. String ("Gs://apache-beam-samples/shakespeare/kinglear.txt") String Getinputfile (); void Setinputfile (String value);/** * Set the output path of the result file, or specify the output file path in the Run Settings option of IntelliJ idea or on the command line, such as./pom.xml */        @Description("Path of the file to write to")@RequiredString GetOutput ();    void Setoutput (String value); }/** * 5. Run the program * /public static void Main (string[] args) {wordcountoptions options = Pipelineoptionsfactory.fromargs (args). Withvalid Ation (). As (wordcountoptions.class);        Pipeline p = pipeline.create (options); P.apply ("ReadLines", TextIO.Read.from (Options.getinputfile ())). Apply (NewCountWords ()). Apply (Mapelements.via (NewFORMATASTEXTFN ()). Apply ("Writecounts", TextIO.Write.to (Options.getoutput ()));    P.run (). Waituntilfinish (); }}

3. Support Spark,flink,apex and other big Data data framework to run the WordCount program.Full project GitHub source (recommended, note the pom.xml success of module loading, in the tool to develop big data programs, facilitate debugging, development experience better)3.1.intellij Idea (Community edition) in the Spark Big Data Framework run pipeline Calculation program

• Spark Run

  • Set VM Options

    -DPapex-runner

  • Set programe arguments

    --inputFile=pom.xml--output=counts

3.2.intellij Idea (Community edition) Apex,flink and other supported Big data framework can run WordCount pipeline calculation program, complete project GitHub source code

• Apex Run

  • Set VM options

    -dpapex  -runner   

  • Settings programe Arguments

    - - inputfile=pom   xml  - - Span class= "hljs-comment" >output=counts   

• Flink run and so on

  • Set VM Options

    -DPflink-runner

  • Set programe arguments

    --inputFile=pom.xml--output=counts

4. Terminal operation (Terminal) (not recommended, the first download process is slow, the development experience is poor) 4.1. The following command is a downloadofficialSample source, the first run download slower, if failed to run more than a few times, (recommended download, complete project GitHub source code) directly with the above interpretation in IntelliJ idea run.

mvn archetype:generate       -DarchetypeRepository=https://repository.apache.org/content/groups/snapshots       -DarchetypeGroupId=org.apache.beam       -DarchetypeArtifactId=beam-sdks-java-maven-archetypes-examples       -DarchetypeVersion=LATEST       -DgroupId=org.example       -DartifactId=word-count-beam       -Dversion="0.1"       -Dpackage=org.apache.beam.examples       -DinteractiveMode=false

4.2. Package and run

mvn compile exec:java -Dexec.mainClass=org.apache.beam.examples.WordCount      -Dexec.args="--runner=SparkRunner --inputFile=pom.xml --output=counts" -Pspark-runner

4.3. Successful run result 4.3.1. Display run successfully

4.3.2.WordCount Output Calculation results

Apache Beam WordCount Programming Combat and Source code interpretation