Kafka Project Practice

Code practice for real-time statistics of user log escalation 1. overview

　　Video tutorial for this course Address: coding practice of Kafka real-combat project

This course I based on the user real-time reporting log cases, with everyone to complete the coding of the KPI, the Production module, consumption module, data persistence, and application scheduling, through the demonstration of this series of processes, Allows you to master the coding and scheduling processes of the Kafka project. Below, let's start by previewing the lessons that are included in this course, respectively:

Next, we begin the first lesson of learning: "Data production realization"

2. Content 2.1 Data Production implementation

This class is mainly to demonstrate the implementation of Kafka data production code, in front of the cluster environment, complete the Kafka data production function, as well as some considerations, for us to write
The consumption code is ready for you to master the code implementation of Kafka data production.

To practice the content of this lesson, we need to design two knowledge points, respectively:

Then, we start from a knowledge point to practice, the practice of data Production module contains the content, there are the following points:

First of all, the 1th is: The project works of the file configuration (POM) and then the cluster link information configuration (why this link information is configured in the configuration file, the reason is that these link information separate to a configuration file, for our later maintenance, such as: The late to add new node information, Or remove a corrupted node information, we can easily and quickly modify the node information in the configuration file, we do not have to operate our business code. Please refer to the video operation for detailed demonstration details.

After demonstrating the content of the data generation module, let me take you through the implementation of the flume to the Kafka module, as follows:

The above is the relevant content of this lesson demonstration, which contains the relevant information configuration, data collection process demonstration.

2.2 Data consumption implementation

This lesson to you to demonstrate Kafka data consumption code implementation, in front of the Kafka Project project we created on the basis of the completion of the consumption code, as well as writing the Storm code consumption Kafka data needs attention to detail, through this class so that you can master the data consumption code implementation.

So, let me give you a list of the core points of knowledge that are covered in this lesson, as follows:

Below, we begin the practice of the first core knowledge point, and the practice contains the content as follows:

1. Information configuration for the Storm cluster: This section contains the dependency link information for the cluster.
2. Dependent file selection: Here we write Java code to implement related functions, we need to select the dependent jar package to assist us to complete the coding work.

Next I'll show you how to encode the implementation of this part, that is, the implementation of the content of the Kafka to the Storm module, which covers the following:

Detailed demonstration details, you can refer to the video operation, view address: "Data consumption realization"

2.3 Persistence of data

The content of this article is about how to persist the data after consumption (that is, the KPI result of our statistics), on the basis of the preceding data consumption realization, and to persist the statistic result to the Redis cluster or DB through streaming calculation, so that we can realize the code of data persistence.

So, next, let's take a look at the core points of knowledge involved in this lesson, as follows:

Below, we begin the practice of the first knowledge point, implementing the content contained in the base Layer code module, as follows:

1. Realization of the idea: before implementing this part of the function, we need to understand one of its implementation ideas, as shown on the right: here, we in the Storm calculation module, the corresponding KPI statistics, do the corresponding persistence, here we can choose to persist to the DB Library of our choice, We persisted to Redis and MySQL, and then we follow this idea.
2. Before we do this, we need to prepare the code for the DAO layer, which is the function of interacting with the db.
3. Next, I'm going to show you this part of the content.

Below, we will implement the relevant content of the storm statistic stored to the DB, and the module contains the following content:

1. Implementation ideas: Similarly, in the realization of this part of the function, we also need to know where to persist the results of our statistics. As shown in the image on the right: we are in the bolt, when our KPI indicator is completed, we can call the corresponding storage code to persist this part of the statistical results.
2. After we have a clear idea, we are going to implement this part of the warehousing process.
3. I'm going to show you this part of the text.

Detailed demonstration details, you can refer to the video operation, view address: "Data persistence"

2.4 Application Scheduling

This section will introduce the development of the application package deployment to the server, through the submission of topology to storm cluster, the implementation of storm consumption program deployment, let everyone master the package deployment of the project and scheduling process. Below, let's take a look at some of the core knowledge points that are involved in practicing this lesson, as follows:

Next, we begin the practice of the first knowledge point. The content that is contained in the package is as follows:

1. The first is the packaged way process, as shown in:
2. Using MAVEN packaging, this project works with the MAVEN structure, where we use the MAVEN command to package the corresponding project.
3. I'm going to show you this part of the content.

Let's take a look at how we can deploy our packaged applications to the storm cluster and run the corresponding tasks. Implement the content that the module contains, as follows:

1. Realize the idea. As shown: Here we want to clear its various stages of responsibility, we in the development phase, in order to develop the convenience and debugging convenience, we can use the local submission, as before, we show you, directly in the IDE, submit the corresponding topology can. In the early production environment, we need to rely on the cluster, using distributed ideas to run our task, so we need to use the cluster submission, where the task is submitted to ensure that the storm cluster is functioning properly.
2. Then the next thing is to implement the appropriate submission process.
3. Next, I'll show you this part.

Detailed demonstration details, you can refer to the video operation, view address: "Application scheduling"

3. Summary

This course we have coded the project indicators, and guide you to code to achieve the corresponding module functions, and to help you to submit our development of the application and other knowledge, should grasp the knowledge:

4. Concluding remarks

We have this knowledge as the basis for our work in the future, the development of similar real-time statistical projects become more comfortable, more handy.

If this tutorial can help you, I hope you can click in to watch, thank you for your support!

Reproduced please specify the source, thank you for your cooperation!

Video tutorial for this course Address: coding practice of Kafka real-combat project

Kafka Project Practice