Apache Flink Docker-compose Run trial

Apache is a streaming framework that officially provides Docker mirroring, and also provides instructions based on the Docker-compose run

Docker-compose file

version: "2.1"services:  jobmanager:    image: flink    expose:      - "6123"    ports:      - "8081:8081"    command: jobmanager    environment:      - JOB_MANAGER_RPC_ADDRESS=jobmanager  taskmanager:    image: flink    expose:      - "6121"      - "6122"    depends_on:      - jobmanager    command: taskmanager    links:      - "jobmanager:jobmanager"    environment:      - JOB_MANAGER_RPC_ADDRESS=jobmanager

Run

docker-compose up -d

Effect

Writing a simple job

Using Maven

• Scaffolding generation

  Follow the prompts to enter information such as group.

mvn archetype:generate       -DarchetypeGroupId=org.apache.flink       -DarchetypeArtifactId=flink-quickstart-java       -DarchetypeVersion=1.6.0 - JOB_MANAGER_RPC_ADDRESS=jobmanager

• Modify Code

Default Project Structure ├──pom.xml└──src└──main├──java│└──com│└──dalong│└──app│├──ba Tchjob.java│├──streamingjob.java└──resources└──log4j.properties Add a technology after the batch is added to the project structure ├──pom.x        Ml└──src└──main├──java│└──com│└──dalong│└──app│├──batchjob.java │├──streamingjob.java│└──util│└──wordcountdata.java└──resources└──log4j.pr Operties Code Description: Batchjob.javapackage com.dalong.app;import org.apache.flink.api.java.executionenvironment;import Org.apache.flink.api.common.functions.flatmapfunction;import Org.apache.flink.api.common.functions.reducefunction;import Org.apache.flink.api.java.dataset;import Org.apache.flink.api.java.executionenvironment;import Org.apache.flink.api.java.utils.parametertool;import Org.apache.flink.core.fs.filesystem.writemode;import Com.dalong.app.util.wordcountdata;import Org.apache.flink.util.collector;publicClass Batchjob {public static class Word {//fields private String word;      private int frequency;         Constructors public word () {} public word (String Word, int i) {This.word = Word;      This.frequency = i;      }//Getters setters public String Getword () {return word;      The public void Setword (String word) {This.word = Word;      } public int getfrequency () {return frequency;      } public void setfrequency (int frequency) {this.frequency = frequency;      } @Override Public String toString () {return "word=" + Word + "freq=" + frequency; }} public static void Main (string[] args) throws Exception {final Parametertool params = Parametertool.fromargs      (args);      Set up the execution Environment final executionenvironment env = Executionenvironment.getexecutionenvironment (); Make parameters available in the Web interface Env.getConfig (). Setglobaljobparameters (params);      Get input data dataset<string> text; if (Params.has ("input")) {//Read the text file from given input path text = Env.readtextfile (Params.get (      "Input")); } else {//get default Test text data System.out.println ("Executing WordCount example with default input         Data set. ");         System.out.println ("Use--input to specify file input.");      Text = Wordcountdata.getdefaulttextlinedataset (env); } dataset<word> counts =//split up the lines to Word objects (with frequency = 1) Tex T.flatmap (New Tokenizer ())//Group by the field Word and sum up the frequency. GroupBy ( "word"). Reduce (new reducefunction<word> () {@Override Publ IC Word reduce (word value1, Word value2) throws Exception {return new Word (Value1.word, Value1.fre QuencY + value2.frequency);      }                  });         if (Params.has ("output")) {Counts.writeastext (Params.get ("Output"), writemode.overwrite);      Execute program env.execute ("Wordcount-pojo Example"); } else {System.out.println ("Printing result to stdout.         Use--output to specify output path. ");      Counts.print (); }} public static final class Tokenizer implements Flatmapfunction<string, word> {@Override public vo ID FlatMap (String value, collector<word> out) {//normalize and split the line string[] tokens = VA         Lue.tolowercase (). Split ("\\w+"); Emit the pairs for (String token:tokens) {if (token.length () > 0) {out.collect (            New Word (token, 1)); }}}}}util/wordcountdata.javapackage com.dalong.app.util;import Org.apache.flink.api.java.dataset;import Org.apache.flink.api.java.executionenvironment;public CLASS Wordcountdata {public static final string[] WORDS = new string[] {"To was, or not to being,--that's the Qu estion:--"," Whether ' tis nobler in the mind to suffer "," the slings and arrows of outrageous Fortun             E "," Or to take arms against a sea of Troubles, "," and by opposing end them?--to die,--to sleep,--", "No more;  And by a sleep to say we end "," the heartache, and the thousand natural shocks "," this flesh is heir To,--' tis a consummation ', "devoutly to be wish ' d. To die,--to sleep;--"," to sleep! Perchance to Dream:--ay, there's the rub; "," for. Sleep of Death What Dreams may Come, "," Whe N We have shuffled off this mortal coil, "," must give us Pause:there's the respect "," that makes CA Lamity of So long life; "," would bear the whips and scorns of time, "," the oppressor ' s wrong , The Proud man 'S contumely, "," the pangs of Despis ' d love, the law's delay, "," the insolence of Office, and the SPU            RNs "," that patient merit of the unworthy takes, "and" when he himself might he quietus make ", "With a bare bodkin?" Who would these fardels bear, ', ' to grunt and sweat under a weary life, ', ' but that the dread of SOM Ething after Death,--"," the Undiscover ' d country, from whose Bourn "," No Traveller returns,--puzzle s The'll, "," and makes us rather bear those ills we had "," Than fly to others so we know not of            ? "," Thus conscience does make cowards of us all; "," and Thus the native hue of resolution ",            "Is sicklied o ' er with the pale cast of thought;", "and enterprises of great pith and moment,", "With this regard, their currents turn awry,", "and lose the name of Action.--soft you now!", "theFair Ophelia!--Nymph, in thy orisons "," is all my sins remember ' d. "    }; public static dataset<string> Getdefaulttextlinedataset (Executionenvironment env) {return env.fromelements (W    ords); }}

• Build

cd  flink-appmvn clean package

• Submit Job

  The generated code is simple

• Run Job
  
• Effect

Resources

https://hub.docker.com/r/library/flink/
Https://github.com/apache/flink/blob/master/flink-examples/flink-examples-batch/src/main/java/org/apache/flink /examples/java/wordcount/
Https://github.com/rongfengliang/flink-docker-compose-demo

Apache Flink Docker-compose Run trial