Internet server Technology--How to learn (middle)

Boys & Girls, our intermission is over, Ha, move the bench to continue listening to Lao Wang technology of the Light ~

(Quietly broadcast: If you have not read the previous student, please pay attention to Lao Wang's simplemain, or pull to the end of this article scan/identify QR code)

In a previous article, we talked about the overall blueprint for Internet server technology. Along this blueprint, we introduce the language, algorithms and data structures, the specific requirements of the framework and how to learn them. Next, Lao Wang intends to introduce a few more basic, but also in the engineering of a large number of use of technology. What exactly is it? We went on to look down. First of all, let's put that blueprint in place (start loading B ~).

Smart classmates must have found that this article to introduce: operating systems , databases and networks , in our blueprint has accounted for two layers. What do these two layers represent? Lao Wang looks like this: the first layer is the system itself, the first layer is based on the application of the system. For example, the operating system covers both the operating system itself and operating system-based use. What are the requirements of these two layers, and how to learn them?

• Operating System ( Operating System )

What is the first reaction when you hear the operating system?

Ten years ago, my first reaction was: win98 (is not very old memory, ha ha ~);

Now, my first reaction is: Android or iOS?

What about you?

We are a code worker, born with the mission of breeding offspring (biological mission) and creation Program (Social mission). Our offspring grow on this earth, and our program runs on the operating system. So, to get our program running faster and better, we need to understand the operating system as well as the environment we generate.

Our study of the operating system can be divided into two aspects to learn: one is a specific operating system (such as: Windows, Linux, etc.), and the other is the operating system principle. Depending on the level of competency, the corresponding requirements are different.

[Primary Ability]

To get the program running, we need to be proficient with an operating system. Think about it, are we already familiar with the operation of Windows? What are the common operations? What do we do to run a program?

I believe that 99.99% of the handlers, the answer to the above question: your question is simply insulting my IQ (Lao Wang is ready to be beaten)

All right, old cuhk answer the above questions. We have to be familiar with Windows, which is actually a common operation, a way to understand how his files are managed, to know the state of CPU and memory, and so on. Common operations include the following:

1. Creation, opening, processing, and closing of directories and files (remember that "My Computer" in the top left corner of the desktop?) ）；

2, download, install, run a program ("Next", "Next" ... "Finish", "Double click Open");

3, observe the CPU, memory, disk, network situation (taskbar right click on "Task Manager").

If you are using Windows as a server, basically the requirements for the operating system are like this. However, unfortunately, most of the Internet servers today use Linux as a server, the code workers need to learn the above operation of the corresponding Linux command (Lao Wang has recently taught the family to lead the use of Linux, preparing to write a Linux starter manual ...). ), the specific is:

1, mkdir, CD, Vim ...

2, make&& make install, yum Install, apt-get install,./run ...

3, PS, top, free, DF, netstat, vmstat, iostat ...

If there is no Linux foundation, it is not very difficult to find?!

In fact, you think of Linux as a text version of Windows, this thing is simple. First the Windows operation translated into English, and then abbreviated, is tall on the black Screen + command (especially in front of the sister play black screen, is the ultimate pack B). Like what:

A, new directory,make directory-mkdir;

B. Double-click to open a directory,change Directory-> cd;

C, disk surplus,DISK free->df.

So, the requirement for this level of competency is to use Linux just as you would with Windows. Learning is not so boring to remember the command, but in the way of Windows operations to analogy, which can cover about 80% of the operation. There are some special, need special to remember.

In addition, the Linux command has a very troublesome thing, is a bit more parameters, what to do? Linux commands generally provide two kinds of help:

1, command--help (such as: CD--help), will provide you with a brief parameter hint

2, the Man command (for example: Men CD), here is not the guy, but the manual of the abbreviation. Help you understand this command in a comprehensive way.

As long as there are these two artifacts, most of the problems can be fixed. If not, use another artifact:

3, Browser input: www.baidu.com Enter, search box input: Linux command ...

In addition to the basic commands, there are a few things to learn: Pipeline (|), input redirection (<), output redirection (>). In general, how about this, is it easier?

[Intermediate Ability]

When I interview other programmers, I often like to ask a question: When you telnet to your Linux server, found that the execution of the command is very stuttering, how do you analyze what the problem? And how to solve it? (This problem actually extends to other server operating systems such as Windows)

To ask this question, in fact, is to examine the interviewer's familiarity with the operating system, some of the original rational things to see, and how to solve him.

When you take part or all of the server operations, or need to do related development on the operating system. At this time, not only basic commands, basic operations, but also some of the following capabilities:

1, write the script: In addition to understand the commonly used commands, we also need to use some advanced commands, such as: awk, sed, chown, chmod, find, sort, and so on. At the same time, it is necessary to understand some of the shell keywords, such as: if, for, while, read, and so on, the combination of these commands, write some slightly logical script, convenient for the machine and program management;

2, understand the principle: we do not know the operating system, but also know why. Understand the operating system some of the original rational things. For example, how processes are created, what are zombie processes, how files are created, what is inode, how data is stored on disk, what is seek, how memory is allocated, and so on. This allows us to maintain the normal operation of the server, analyze and solve some problems in the operating system.

3, will program: Understand the operating system provides the API, can be the original rational east to the program, write the function code we want, and compile, link and other operations. For example, we monitor the operation of a machine, we need to write the corresponding monitoring program, CPU, memory, disk and so on real-time analysis, if there is a problem, remind the management personnel. (Many high-level programming languages now provide virtual machines, which makes the program a good cross-platform capability.) This actually helps a lot of programmers out of misery. However, if you are writing a high-performance server program, you may still need to know about C + +, even assembly language, and understand the API provided by the operating system.

These are basically half experts, enough to deal with most situations.

How do you learn that?

For commands and scripts, Lao Wang used to like Ls/sbin and Ls/bin, to see what commands, and then go to the Internet to check, see what they are doing. For some of the more well-known commands, use Man, look at their detailed usage, and write a script to actually use it.

For the original rational East, Lao Wang suggested that we look at the operating system principles related books, above all have a more detailed explanation: CPU management, memory management, file management and so on.

For programming, if it is a Linux server, recommend everyone a treasure: Apue, "advanced programming in the Unixenvironment", Chinese name: "UNIX Environment high-level programming." Lao Wang has probably read it 2-3 times and has benefited from every reading. But light reading is not possible, it is best to follow the example of writing, to understand. The meaning of reading is not great; if it is a Windows server, Lao Wang wrote some windows program more than 10 years ago, was still in the undergraduate 1-2 grade, that time still used VC6 (that is how old memory), read more than half "windows programming", feel good. I don't know how it is now, so Lao Wang doesn't recommend it.

[Advanced capabilities]

If you are an operational expert, or a developer of an operating system. You need to have an in-depth understanding of the operating system itself. This includes two levels of Dongdong:

1, the operating system principle has a more in-depth understanding: Processor management, memory management, file management, equipment management, etc., after the image, covering the process, pagination, file storage and so on, let you look at the operating system from an abstract perspective.

2, deep One operating system: can put the operating system principle of the east to a certain operating system. For example: implementation of Linux file management. Or to customize and recompile an operating system based on business requirements.

Because Lao Wang just learned the operating system principle, the general principle has a certain understanding, do the bottom of the development of the time involved in some things, but most did not go deep into practice, so this part does not give too good advice, can only be recommended to read the relevant books. If there are experts in this area, also welcome to give advice, we learn and progress together ~

Operating system is our code workers must rely on the living environment, every day will face. So if you want to engage in the development of Internet services, it is best to use at least two systems of Windows and Linux. A good desktop office environment for you, a good program for you to run the environment.

• Database ( Database )

Before talking about database skills, I want to ask you a question: Why do you have a database? Or you can change the question: What's the impact of having no database?

Before answering this question, Lao Wang told a story. When I first entered Baidu (about 07-08 years), Baidu (at least I am in the large department) also basically did not use the database, all the system developed in C + +, in addition to consider the logic level of the East, Also consider how to write the logical data generated by each system into a structure stored to a disk file (serialized write), and how to read the structure data (serialized read-in) from the disk file. In order to develop a functional module, it needs to be in months, which in mobile interconnection today, is simply unbearable slow. And what do we do now? Write logic in high-level languages such as Java/ruby, store data in a database, and develop a full-featured system for a few weeks (if it doesn't work, it may take a few days).

Yes, the database is to help us solve the problem of data storage, he provides a common data storage and reading scheme. can help us to solve most of the data storage problems (some of the system database may be uncertain, or the cost is relatively high, later on, and then share with you), greatly improve the speed of our development.

Our commonly used relational databases, such as MySQL, SQL Server, DB2, Oracle, PostgreSQL, etc., all include: SQL operations, transaction management, file storage, index management and other functions (different databases in the implementation will have functional differences, but the large Wannna is consistent). They provide these features enough to meet the needs of our daily development (it is completely the savior of the code workers, if there is no database, it is estimated that the code workers have been killed by the product manager, ha ha ~). Later, a few years ago began to popular non-relational databases, such as MongoDB, Redis and so on, these NoSQL databases to lightweight design, fast query efficiency, in today's Internet research and development also played a very important role (for example, the rapid change of demand for experimental products, MongoDB is used in document storage to avoid the constraints of traditional relational database, Redis provides more data structure support and so on.

In general, we use a combination of NoSQL and Rdbs in large-scale data access and storage scenarios. Often, a fast, lightweight, non-relational database is used as a temporary data access medium, for example, Memcache and Redis are often treated as caches for a large number of duplicate requests. Relational databases are used as persistent storage with high consistency requirements.

Therefore, for different application scenarios, we are not the same requirements for the database.

[Primary Ability]

This level of competency requirements, is to meet the daily business development needs, straightforward point is: can use! So:

1, either use a non-relational database: will use the program to access the common set, get, delete, modify and other interfaces;

2, or use a familiar relational database: will be commonly used SQL to do interpolation and deletion; The database is accessed by the program.

Of course, the above premise is: Learn to install and configure some kind of database. These requirements are the code workers to solve basic food and clothing the most basic

[Intermediate Ability]

If you are a company's service-side technology leader, you need to have a more in-depth understanding of the database in order to deal with the data volume, data consistency, query and submit the number of concurrent requests and so on. In addition to the basic operations, you need to have a more in-depth understanding of database principles and a database. It is possible that both non-relational and relational types need to have some involvement.

1. Non-relational database: How did he achieve it? What special features are available to help us meet different needs? What are the bottlenecks and problems? After the amount of data is large, how to ensure the consistency and availability of data? And so on

2, relational database: Need to understand the SQL operations, transaction management, file storage, index management, such as a few of the basic principles of the basic can answer the following questions: how each SQL request to do query optimization? How is the index designed? And the kind of index and how it works? How are transactions managed (such as MVCC)? How is the data stored to disk? Wait a minute.

So how to learn it? For the implementation of the database is not complex, the best way is to read the source code, for the original reason, the need to look at the algorithm data structure, operating system principles of knowledge to supplement (such as: B-tree and + + tree, hash algorithm; Read, write these IO calls, etc.). Another point, computer science is a practical science, light reading is not possible, but also need to write code practice. So, if there is a chance, you can use the file to implement the storage. After writing, many problems will be more in-depth understanding (such as: Fwrite is there a cache?) Does write have a cache? What is byte alignment? What is mmap, directio, etc.).

In addition, you may want to study the configuration instructions for different databases in detail. Configuration parameters are different, the performance of the database can vary greatly. For example: How big is the memory adjustment? What is the Read timeout configuration? How often does the data brush back to disk? Wait a minute

[Advanced capabilities]

This is the request of the expert level. Very proficient in database principles, a database system in depth. In addition to meeting the development requirements, you can also make code changes to the database to patch, the database to improve the function, and so on.

Lao Wang thinks he can not reach this ability requirement, so this aspect does not suggest much. But in the same time in the Baidu cohabitation Honey friend, now is the Big Ben MySQL, last year asked him to hundred words to do technology sharing.

Internet service side of the biggest technical difficulties, is the storage of data (Lao Wang chose Baidu is a very important reason, is his own work in the time, to solve the large number of visits, data read and write slow problem). Data storage is divided into special data storage and universal data storage. One of the most common data stores, the best solution now is the database. Therefore, to learn the database technology is the most serious of the Internet service technology.

• Network ( Network )

The so-called Internet Internet, the most important of course is this network. Everyone to the TCP/IP, HTTP these nouns, estimated to listen to the ears are cocoon.

For the development of Internet server, we generally value three aspects: network protocol, Network service model and network security. Network protocol is the basis of all our work, is the basic theory of computer networks, the Network service model can enable us to have more efficient service capabilities, and network security is to protect our service security Foundation.

For different characteristics, the requirements of different levels of competence naturally vary.

[Primary Ability]

1, protocol Knowledge: Know the principle of network stratification, understand the common network protocols, and their proficiency in the application. You need to understand the common protocols:

A, HTTP protocol. The HTTP protocol is the foundation of the Internet, so it's a must-see. What is a header, what is a cookie, a common return code (2XX, 3xx, 4xx, 5xx), what is post, get, and so on. Use the tool to send HTTP requests, or to send HTTP calls using a program.

b, TCP/IP protocol. This also naturally needless to say, as a code worker, definitely should know what is an IP address? What is a port? How do I establish a connection? Wait (if this does not understand, how to help the girl to repair the computer, right?) ）

2, Service Model: If it is a common Web service, you need to understand the common Web server, will be built, will be configured, will be used. Ensure that our users can successfully access the service. For example: Now the popular Nginx (or previously popular Apache, Lighttp, etc.). If it is the socket service, you may need to focus on some other servers or components (such as Java Mina, Netty, etc.), will be configured, will be used.

3, network security: Understand common security threats, such as: DoS, DDoS, XSS, SQL injection, and so on. When writing code, it is possible to proactively circumvent common coding vulnerabilities (such as HTML escaping, SQL escaping, and so on).

[Intermediate Ability]

The requirement for this competency level is primarily to have a comprehensive understanding of the network, and to write high-efficiency network programs that can guard against some common cyber attacks. Therefore, it is necessary to fully understand the following knowledge:

1, agreement knowledge: for the development of Internet services, the focus is to focus on TCP/IP and HTTP protocol, the specific knowledge covers the following:

A, how the network stack is layered, and the responsibilities and working principles of each layer, and the package format and application scenario for each layer of common protocols;

b, TCP, IP protocol: The format of IP packets, protocol routing rules, IP address related content (gateway address, broadcast address, subnet mask, a-d class address, etc.); TCP three handshake, graceful close, sliding window, timeout retransmission, etc.

C, Http, HTTPS protocol: how the two protocols work (HTTPS can be found before the old Wang wrote an article "https What is a thing?" "), and some in-depth usage, such as keep alive, range, etc.;

D, some other technologies, such as: Network address translation (NAT) and so on.

2. Service Model:

A, will write the basic client-server code;

b, to understand the common network model, it is best to write the relevant model code. For example: SELECT, Epoll;

C, the role of common setsockopt options: So_rcvtimeo, So_sndtimeo, so_reuseraddr, So_linger and so on.

D, understand the common webserver architecture and working principle, such as Nginx, Apache;

3. Network security:

A, understand the principles of common attacks, and how to prevent and deal with. For example: The cause of the SYN semi-connection attack and today's solution.

b, the use of common network tools and methods to avoid network risk. For example: Configuration and use of Iptables, database authorization management and security configuration, and so on.

For a more comprehensive understanding of this part of the network, we recommend that you read the classic "TCP/IP Details" trilogy, as well as read the common Web server source code, but also pay attention to wooyun.org above the relevant knowledge.

[Advanced capabilities]

At this stage, it is basically an expert in one area. Either the network protocol is very proficient, or the network service model is well known, or network security experts. Specific to:

1, fully understand the network protocol, and can write protocol code;

2, or write efficient large-scale network server;

3, or the network security attack and defense at a very familiar.

When Lao Wang was reading, he wrote the ICMP protocol, but did not hit the ground with the UDP analog to implement TCP. After work, take time to see part of the Apache code, java Mina, thrift code, changed some of the Tomcat code, the Java implementation of the reactor model code, but there is no chance to apply to large-scale network access. However, for the network attack and defense do relatively little, no experience.

Lao Wang came to an experience is that there is a chance to read more books, more read the source code, more thinking, more tossing ~ life endless, tossing not only.

= = = Break Split line = = =

Well, the story of Lao Wang is finished today, what do you think? The next article compared to the previous and medium, more emphasis on the actual engineering problems encountered, so the actual operation is more strong, the content more paste, please pay attention to the old Wang next a nonsense bar ~

Narrating: Lao Wang is now basically an article every week, a friend asked Lao Wang why not every day out? The reason is this. May be a lot of friends have found, Lao Wang's article is generally very long (seems probably probably 4000-8000 words, so thank you have patience to read so lengthy text), pure code word about to get a few hours. At the same time working days also have to work overtime, so generally only weekends have time to code word. And Lao Wang is in charge of everyone, the attitude of responsibility to the heart, will be repeated review and changes (this article has been modified at least 4 times). So Lao Wang try to ensure that a week out of an old Wang himself feel Good article, I hope you understand ha ~

Still not paying attention to the old king? Add it quickly:simplemain

or sweep code/long Press identification QR code:

Come on, look at the beauty map. This is the 2012 Lao Wang to Manzhouli-Hulunbeier-Genhe-Eerguna shot of the beautiful scenery. Lao Wang walked through about 20 provinces, which is the place to be Top3.

Internet server Technology--How to learn (middle)