"Node" records the beginning and completion of a project--pipeline_kafka streaming database management project

Objective:

I always believe that the effect of learning 80% from the summary, and even all of them. The benefit of summing up is that you can turn out your past, point out the shortcomings, see the future of improvement methods, good summary can make the knowledge leap forward, so in the spare time, deployment project, summed up.

Schema processing:

　　I. BACKGROUND

The company's data volumes were huge, and the kaleidoscope of leaders immediately used the streaming data architecture of Pipeline_kafka. In fact, before taking over the project, as an application development, I am not very clear Kafka is what ghost, there is stream, transform, and so on a series of nouns, sounds very tall, but very ignorant than =-=.

The goal of the leadership is to make a data flow management system, which is to correlate postgresql+kafka+ flow, manage Kafka's association, Flow data rules, and transform data operations by means of a platform, simply by integrating data Engineer's usual SQL operations. Improve productivity with a visual platform.

　　Second, the front-end structure

front-end frame, choose the react.js, slightly use a little bit to feel the react as a view of the charm of the library, it will become more elegant front end. Because React.js can only handle the view layer, and the specific controller I just use jquery as a supplement, the project's volume is small, change more, using jquery is the best choice.

List the technologies used:

①react.js

②jquery

③react-tools Packing JSX

④sass CSS Modular Processing

Look at the front end effect Click ""

Because the project is simple enough, react is also a shallow use of a bit, and did not use the ES6, nor used to webpack packaging, as one side of the project, side into the door, the idea of simple, practical, and look at the code structure:

Take the homepage as an example:

　　First, a common index.html file introduces the required file resources:

　　

　　CSS processing, using SASS's @import for modular loading:

　　

The benefit of this is that it can be loaded on demand, and a index.scss can resolve all dependencies and reduce the number of requests as they are introduced.

See React again:

Component.js Package All components, page common

　　

　Index.js hosts the main components of the page, interacting with each other.

After running all the pages, deploy to node to open React-tool, remove browser.js for app development.

Personal feel that this front-end structure of the bright spot is that the front-end structure is clear, organizational logic is more understandable, suitable for use as a transition from traditional development to automated development of the excessive structure, also suitable for the development of the environment special, using NPM misery of the masses, such as my company, work computer is completely isolated from the Internet, Packaging with Webpack is extremely inconvenient. Another project if not sure in the future will fall to which programmer's hand to maintain, using webpack packaging has the potential to cause trouble, after all, to maintain the code of the people, he may be a Java development ...

Resources:

React Jsx compiled as JavaScriptReact Introductory Example Tutorial

Third , server-side structure

　　The Classic Express framework MVC structure that gets the front-end code to node, the entry file looks like this:

The PAGENAME variable is determined by router's render, so the optimization point is that the page is managed by react, the independent JS file can modify the performance stronger, and also strengthens the logic.

Classic Express Entry File

Router Routing control:

Resources

Express development and deployment best practices

Nodejs Advanced: How to play rotor process (child_process)

Promise of the latest technology stack of node. js

Nodejs

　　Projects require cross-domain, token-based login verification

　The project needs cross-domain requests, so how to telnet into the problem, Google for a long time to use the token mechanism, to generate tokens this step did not use the third party package, but played the next spear, and see the code implementation:

　

The token is hung on global, settimeout time-lapse deletion, and then get token comparison according to name, cross-domain set credentials get cookie, or through headers to obtain token, security performance is good, Later I hung it in another process, get and set to manage using another process.

More I will open another article slowly said. 〓

Data:

　　First, Kafka

　　Nodejs Connection Kafka I use the Node-rdkafka component, this component can only be compiled and used in Linux environment, Node-rdkafka git address

The reason to use it is that it supports SASL certification, this component has been looking for a long time to find, the specific connection code in the work environment can not get out, here also does not show the introduction, GitHub has detailed instructions.

　　Second, PostgreSQL

　　Using PG components to connect PostgreSQL, packaging and adding and deleting methods:

　　

Third, other

　　More details, because the space is too large, put in other articles to tell, here a short summary of the main structure of the project.

Summarize:

Overall, the project is not big, can play the pattern is not many, among them the technical points, point through also handy, Bo subordinate to the mission to the deployment, took 10 days to develop the completion of the time, feel that there are a lot of shortcomings in each place. Summary written on the blog and you discuss.

Any suggestions, please leave a message.

Need to know more detailed knowledge of the points, you can contact me.

"Node" records the beginning and completion of a project--pipeline_kafka streaming database management project