Apache Flink-Basic API Concepts

• The Flink program is a general program for implementing distributed set conversions. The collection was originally created from the source. The receiver (slink) Returns the result, and the receiver can write the data to a file or to a stdout. Flink can be run in variousenvironments (contexts), local JVM or cluster.

1. Datasets and Data streams

• Flink with special classes and DataSetDataStream来表示程序中的数据。可以认为他们是可以包含重复数据的不可变数据集合。在DataSet中数据是有限的，而在DataStream中数据是无限的。
• These collections are different from the set in Java, they are immutable, once created can not be changed, and can not simply check the elements inside.
• The initial collection is created by adding a source to the Flink program, and the new collection is obtained by using the API method (e.g. map , filter ).

2. Anatomy of a Flink program

• Each program contains the same basic parts:

1. Get an execution environment (execution environment).
2. Load/create initial data.
3. Specifies that the data be converted.
4. Specifies the location where the calculation results are placed.
5. Triggers the execution of the program.

• Streamexecutionenvironment is the basis for all flink programs. Can be obtained by the following static methods:

  int port, String ... jarfiles)

  Usually only need to use the Getexecutionenvironment () method, because it will do the right thing according to the environment: if you execute your program on the IDE or as a normal Java program, it will create a local environment that will execute the program on the local machine. If you create a jar file from your program and invoke it from the command line, the Flink cluster Manager will execute your main method and Getexecutionenvironment () will return an execution environment that executes the program on a cluster.

• Used to specify a data source, the execution environment has several methods to read from a file: You can read it line by row, like a CSV file, or use a fully customizable data entry format. To read the order of a text file, you can use:

  Final Streamexecutionenvironment env = streamexecutionenvironment.getexecutionenvironment ();D Atastream< string> Text = Env.readtextfile ("File:///path/to/file");

Apache Flink-Basic API Concepts