Linux kernel architecture and the bottom--reading notes

Pipe symbol "|" in Linux The role

Command format: command a| command b, which is the correct output of command 1 as the operand of command B (apply someone else's picture)

1. For example: PS aux | grep "Test" looks for test in the results of PS aux.

2. For example: Find. -name "*.txt" | Xargs grep "Good"-n--color=auto the results of find as parameters into grep, which is the search for good keywords inside those files.

Note: In this example, Xargs splits the long list of files created by find into multiple substrings,

such as "" Find/path-type f-print0 | xargs-0 RM

Xargs splits the long list of files created by find into multiple substrings, and then calls RM Xargs for each substring may be mistaken, and if you need to handle special characters, you need to use the-0 parameter for processing. option Explanation
-0: When Sdtin contains special characters, think of it as a general character, like/' space ', etc.

2.1 Linux File system

Linux sees everything as a file, a point that everyone knows. I want to say that, in addition to the traditional ext file system, Linux in the abstract different resources when there are a variety of file systems, are from the requirements and use, how convenient how to come, such as the proc file system is the abstraction of the process, so that the change of the value of the corresponding process can directly change the behavior of the process. For example, for the PTS device with remote SSH login, Linux has a corresponding devpts file system.

2.2 Rights Management for Linux

Linux's-rwxrwxrwx Rights Management is well known, in fact, Linux itself is aware of the limitations of such a permission to manage. First of all, rwx Rights Management is based on the user and group, and only roughly divided into owner|group|other these three categories, can not be more fine-grained division. For this reason, Linux is currently managed by default with an ACL (Access control List), which is the ability to provide finer-grained user and group management, such as a way to identify which user can have what permissions. The following example

?

123456789

getfacl abc# file: abc# owner: someone# group: someoneuser::rw-user:johny:r-xgroup::r--mask::r-xother::r--

While SELinux provides permissions management that is not based on user and group, SELinux is an application-based, what kind of application can use what resources, for these resources this application is capable of, this is the SELinux management way.

2.3 Service on Linux

The service organization on Linux is very clear and, of course, traditional. /etc/init.d/contains all the service startup scripts, the corresponding binaries in/usr/bin,/usr/sbin,/usr/local/bin and other directories, the general configuration file under/etc/app_name, There is also a chkconfig tool to manage the service that needs to be started under each runlevel. Such conventions make it very convenient for administrators to configure and use them. The Linux standard service will log log to the/var/log/messages, so that the system administrator does not need to flip through the various logs, directly in the/var/log/messages can find most of the log to determine whether the current system is normal. What's more, SYSLOGD is replaced by RSYSLOGD, you can send the contents of/var/log/messages through UDP to the remote with Professional log analysis tool for analysis. We need to learn these excellent programming habits and techniques for service on Linux.

3 disk

Based on the requirements in $ $, the main is to create LVM under Linux, as well as some basic disk operations.

• Df-lah to view disk usage
• Fdisk-l View the hard drive inserted into the disk drive; SD (A,B,C), where a is the first disk, B is the second disk, and three-in-one represents the primary partition on the disk, up to 4. Use Fdisk to create partitions from disk and format them.
• LVM (Logical Volume manager), is mainly to meet the drive can directly write data functions, and will not appear full, and then hung. LVM has several concepts, VG, PV, LVM, format the disk LVM, create PV, create a VG, add the created PV to the VG, and then create LVM in the VG, and then you can dynamically increase the size. Note that the disk is formatted as LVM, but the format of the LV needs to be ext before it can mount. Refer to this article CentOS 6 volume group Mount HDD Tutorial
• MOUNT-T type (EXT4|NFS)/dev/sdxn/path/dir to mount. If you want the reboot to take effect, you must write the mount information to the/etc/fstab
• Disk IO efficiency (IOPS) needs to be viewed with tools such as Vmstat and top.

4 Network

Network pit a lot, need to put network straightened out not a 3, 4 years difficult. Let's start with the basic configuration file of the network, and simply get a quick look at what's on the web. The most difficult aspect of the network should be how to build a reasonable and efficient LAN or metropolitan area network, this need to have professional network knowledge.

4.1 Configuration Files

/etc/hosts private IP corresponds to host name
/etc/resolv.confnameserver IP for DNS
/etc/sysconfig/network where networking= want to have network, hostname= hostname, networking_ipv6= support IPv6 No
/etc/sysconfig/network-scripts/ifcfg-xxx which device= NIC code, bootproto= whether to use Dhcp,hwaddr,ipaddr,netmask,onboot,gateway
4.2 Network design to some of the commands

Router-n View the command of the route, especially to see with G, which represents the gateway, and the expression up with U.
NETSTAT-ANP See all the applications that start with Tcp,udp,unix stream, as well as their status, refer to the Tcp/ip,javasocket simple analysis article.
5 safe

5.1 PAM

Pam, in general, simply needs to understand the line, is a pluggable authentication module. In my opinion, it's a reusable component of the Geek who developed Linux. For example, now there is an app that wants to verify that the current login user has permission to operate a directory, then there is a ready-made module in Pam, the app only needs to include this module, give a configuration file, it can. There is a very good video tutorial on Pam, see here

• Pam is the application used for authentication. Early authentication is coupled with the application itself, and then the authentication is extracted separately and managed through Pam
• /ETC/PAM.D/XXX is an application Pam setting that can be managed with Pam and installed when the application is installed. /etc/security/mmm,/lib/security/pam_mmm is a set.
• In general, Pam is using the mechanisms provided by the Linux system itself to verify

5.2 SELinux

There is also a very good video tutorial on SELinux, please see here

• Getenforce to see if SELinux is enabled
• /etc/sysconfig/selinux enforcing Enable SELinux
• SELinux configures and checks whether it has permissions to operate the "object" (File system), and the normal ACL (RWX) is judged by the owner and its group that the file belongs to. SELinux is the compatibility of the type of the bin and the type of the directory file to determine if the bin can manipulate resources
  

5.3 Firewall

For Iptables is also smattering, so the following is just some excerpts from the study time. In particular, to open the kernel parameter net.ipv4.ip_forward=1, in the/etc/sysctl.conf file, with Sysctl-p to save. The so-called Ip_forward refers to the IP packet forwarding from one iface to another iface provided by the kernel, such as forwarding IP packets from 192.168.1.10 eth0 to 10.0.0.123. Professional firewall configuration is required for professional skills.

Tcp_wrapper need libwrap.so support, all ldd out of the bin, can not be used Tcp_wrapper
Iptables is based on the rule to determine the short-circuit, that is, meet the conditions 1-> execution action1-> end
Iptables-save to see more clearly
It's easier to delete all the rules and add them first. When adding, add a policy and then add a detail rule. In general, what we need to focus on is the input and output of the filter table.
Iptables-a (I) INPUT (Output,forward)-I (o) iface-p TCP (Ump,imp,all)-S (!) Source-d dest-j ACCEPT (Reject,drop), also supports the parameters-dport-sport
6 Tools

A good Linux command reference site

6.1 CPU

• Top pay special attention to load
• PS aux and Ps-ef pay special attention to process status
• Vmstat 1 means capture once per second
• Sar-u 1 View all CPU-related uptime

6.2 Memory

• Free
• Vmstat 1 Note the relationship between the block of swap RAM
• Sar-r 1 Memory utilization
• Sar-w 1 View swap, query for large memory swapping due to insufficient memory
  

6.3 IO

• Lsof-i:port query which process takes up this port number
• Lsof-u username User-opened files
• Lsof-p PID Process Open File

1. Common architecture paradigm of kernel architecture:

• How the Linux kernel communicates on the upper and lower layers
• Transverse systems and longitudinal systems

transverse systems such as Cgroup,proc,sys file system, System call organization, debug system, Core Dump, signal, memory management, etc.
Longitudinal system refers to the specific function module, such as USB function, a USB file operation to go through the core of many layers, that is, file system layer, cache layer, universal block layer, SCSI layer, USB layer. Linux generally divides these layers into 3 large levels, interface layers, functional logic layers, and drive layers.

2. Module support

• Modules are the primary mechanism for Linux to support dynamic feature extensions.
• The module can specify its acceptable parameters when programming, and this parameter is for the user. After the module is loaded, user space can dynamically modify the module parameters via "Echo-n ${value} >/sys/module/${modulename}/parameters/${parm}".
• The meaning of the module mechanism is that it can be loaded and unloaded dynamically.
• Module signature, each module will be compiled from the kernel directory to obtain the version number written to the compiled module, the running kernel when inserting a new module will detect whether the signature is consistent, if inconsistent will not load. You can use Modinfo to view module signature information. The contents of the module signature have two parts: version number, hash signature
• Module programming can be used kernel components: Workqueue,linux Work queue, you can postpone work, sleep, schedule, the system has a default Workqueue kernel thread, but also support users to define their own workqueue, interrupt system and Tasklet, Interrupt affinity can be used to help lock application performance in application operations.

3. Special software mechanism

• UIO, which allows the client to access the device details directly, is a kernel-driven mechanism that is implemented on the client side.
• Vfio, is the software to hardware device memory exposure in the user space support, is the upgrade of the UiO. The user can manipulate the hardware by directly manipulating the memory space of the hardware.
• SYSRQ, similar to the effect of the Windows Ctrl+alt+del key combination, responds to this command as long as the system is not completely locked.

in Linux you can control the closing and opening of the SYSRQ, using echo "1" >/proc/sys/kernel/sysrq to open the mechanism.
In Linux, this series of commands is called "sysrq+ command", SysRq on most keyboards is a secondary function of the Print Screen button, you need to use the ALT key to invoke.
Sysrq+b: Restart the system now
Sysrq+c: Produces a system-level crash dump
Sysrq+d: Displays all locks currently in use
Sysrq+e: Send sigterm to all processes except Init
In addition to the above, there are more than 10 other similar commands

• Other mechanisms, padata,namespace

4, the core data structure of the linked list and hash table

• Linked lists, with elastic fixed properties, can string the data structures that arrive at discrete times so that they can be indexed more easily and do not need to move the previous content.
• A kernel hash table consists of a list of linked lists, each of which is a list of its hash buckets.
• A doubly linked list, whose data fields generally include Prev, next, and data, are easily made into a ring.
• Hlist, with a doubly linked list with only one pointer-sized head (using only one head, to solve the space utilization problem of the hash bucket).
• Scatterlist, a software structure generated at the same time as DMA supports separate multi-block memory, which represents multiple blocks of detached block memory. Linux allows stitching or merging of scatterlist.
• Llist, a list that does not need to be locked, is the data structure used by the kernel for a lock-free operation to improve processing efficiency.

5. Other data structures

• B + Tree, the Linux kernel implements a common B + tree, mainly used in file systems, and of course some file systems are implemented by their own similar B + tree.
• Radix tree, which is used in the kernel to correlate pointers with long integer key values, such as the IDR mechanism, and has a high search efficiency.
• Bit array bitmap, which is a method of storing values in bits, is used by most file systems. For example, an inode bitmap and a block bitmap are used in ext to indicate that the inode or data block of the corresponding ordinal is used. In RAID systems, such as RAID1 's data consistency assurance, a bitmap is checked to detect inconsistencies in two of data.
• FIFO, named pipe, is a very useful tool that the kernel provides to user space, equivalent to a cross-process queue, which provides the native Plug and the ability to mate files. Similar capabilities can also be implemented using Message Queuing or UNIX domain sockets. FIFO file also has a feature, it is a file, even if no one is reading, you can also write content to the file.
• FIFO files are a very effective debugging tool in an interactive shell, using the MKFIFO command to create a FIFO file. In blocking mode, a party keeps the cat file, and if no data is stuck there, the cat reads the data as soon as one party uses Echo to write to it.
• In Linux, the FIFO pipeline does not have a specific data structure, but rather by pointing two file structures to the same temporary VFS index node inode, the VFS index node also points to a physical page. It is best not to use the file's caching function when writing FIFO files, and to write them all at once.
• FIFO is not easy to use, for a serious FIFO scenario, the following conditions of use are required:

Transferring data across processes
Data generation and data monitoring out of sync
Data is deleted as long as it is read, and the second time it is not read again
Need to wait for timeout, permanent blocking, immediate return some or all of these attributes
Need operation and maintenance system to view data flow
There is no way to determine in advance whether there is data in the blocked FIFO to read

Summary of common knowledge points

1.date-s Date Settings

[Email protected] ppp]# date-s 2015-7-5

Sunday, July 05, 2015 00:00:00 CST

Setting the 2.date-d time

[Email protected] ppp]# date-d 3:45:45

Sunday, July 05, 2015 03:45:45 CST

3.HWCLCOK-S Hardware clock synchronization system clock

[[Email protected] ppp]# Date

Sunday, July 05, 2015 00:03:28 CST

[Email protected] ppp]# hwclock-s

[[Email protected] ppp]# Date

Monday, March 28, 2016 03:07:49 CST

4.HWCLOCK-W system clock synchronization hardware clock

[Email protected] ppp]# hwclock-w

[[Email protected] ppp]# Date

Sunday, April 05, 2015 00:00:24 CST

References in 5.bash:

": Strong references

"": Weak reference

": Command Reference

6. Wildcard Practice

(1) Display/etc directory, beginning with a non-letter, followed by a letter and any other length of any character file or directory;

[Email protected]/]# ls-d/etc/[^[:alpha:]][a-z]*

LS: unreachable [^[:alpha:]][a-z]*: No file or directory

/etc/

(2) Copy/etc directory, any file or directory that begins with N and ends with a non-digit to the/TMP/ETC directory;

[Email protected]/]# mkdir/tmp/etc

[Email protected]/]# cp-r/etc/n*[^0-9]/tmp/etc

[Email protected]/]# ls-ld/tmp/etc

Drwxr-xr-x. 3 root root 4096 April 5 00:54/tmp/etc

(3) Display the/usr/share/man directory, all the files or directories that begin with man, followed by a number

[[email protected] etc]# ls-ld/usr/share/man/man[0-9]

Drwxr-xr-x. 2 root root 69632 March 2016/usr/share/man/man1

Drwxr-xr-x. 2 root root 20480 March 2016/usr/share/man/man2

Drwxr-xr-x. 2 root root 495616 March 2016/usr/share/man/man3

Drwxr-xr-x. 2 root root 4096 March 2016/usr/share/man/man4

Drwxr-xr-x. 2 root root 20480 March 2016/usr/share/man/man5

Drwxr-xr-x. 2 root root 4096 March 2016/usr/share/man/man6

Drwxr-xr-x. 2 root root 12288 March 2016/usr/share/man/man7

Drwxr-xr-x. 2 root root 36864 March 2016/usr/share/man/man8

Drwxr-xr-x. 2 root root 4096 September 2011/usr/share/man/man9

(4) Copy/etc directory, all files or directories beginning with p,m,r and ending with. conf to the/TMP/CONF.D directory;

[Email protected] etc]# MKDIR/TMP/CONF.D

[Email protected] etc]# cp-r/ETC/[PMR]*.CONF/TMP/CONF.D

[Email protected] etc]# LS-LD/TMP/CONF.D

Drwxr-xr-x. 2 root root 4096 April 5 01:02/tmp/conf.d

7. There are three data streams for the program:

Input data stream: <--, standard input (stdin), keyboard; stdin=0

Output data stream:-->, standard output (stdout), display; stdin=1

Error data stream:-->, error output (stderr), display; stdin=2

8. Output Redirection Knowledge points:

COMMAND >/path/to/somefile

Overwrite Redirect: Overwrites the original content in the target file;

Example: Touch liwenming

Cat/etc/fstab > Liwenming

COMMAND >>/path/to/somefile

Append Redirect: Append the newly generated content to the end of the target file;

Cat/etc/fstab >> liwenming

9. Pipeline Knowledge Points Summary exercise:

(1) Change all lowercase characters in the last three lines of the/etc/passwd file to uppercase and then output;

[Email protected]/]# Tail-n 3/etc/passwd | TR "A-Z" "A-Z"

Sshd:x:74:74:privilege-separated Ssh:/var/empty/sshd:/sbin/nologin

Tcpdump:x:72:72::/:/sbin/nologin

Lwm:x:500:500:lwm:/home/lwm:/bin/bash

(2) Remove the 6th line of/etc/fstab;

[Email protected]/]# Head/etc/fstab | Tail-1

NI HOA MA

(3) Remove all the files or directories in the/etc directory that begin with p, showing only the first 5;

[Email protected]/]# ls-d/etc/p* | Head-n 5

/etc/pam.d

/etc/pango

/etc/passwd

/etc/passwd-

/etc/pbm2ppa.conf

(4) Tee command, refers to the two-way output, the output screen and then saved to a copy of the file.

[Email protected]/]# echo Lilili | Tee Li

Lilili

(5) Provide the user with the default configuration profile

/etc/login.defs,/etc/default/useradd

(6) Add the user to the group and remove the user from the group

[Email protected]/]# Useradd Ming

[Email protected]/]# Gpasswd-a Ming Li

Adding user Ming to group Li

[Email protected]/]# gpasswd-d Ming Li

Removing user Ming from group Li

(7) Create user Gentoo,uid to 5000, group-based Gentoo, additional group is distro and Peguin;

[Email protected]/]# gpasswd-a Gentoo distor

Gpasswd:group ' distor ' does not exist In/etc/group

[Email protected]/]# gpasswd-a Gentoo Peguin

Adding user Gentoo to group Peguin

[Email protected]/]# gpasswd-a Gentoo distor

Gpasswd:group ' distor ' does not exist In/etc/group

[Email protected]/]# gpasswd-a Gentoo distro

Adding user Gentoo to group distro

[Email protected]/]#

(8) Create user fedora, based on the annotation information as "Fedora Core", the default shell is/bin/tcsh;

Useradd-s/bin/tcsh fedora-c "Fedora Core"

View Results CAT/ETC/PASSWD

[[email protected] etc] #useradd Gentoo

[Email protected] etc]# useradd-g gentoo-g distro Peguin

[Email protected] etc]# Cat/etc/group

Distro:x:1006:gentoo,peguin

(9) Summary of logical operation knowledge points

Logical operation:

Operand: true, false

COMMAND:

0:true

1-255:false

And:

True && true = True

True && false = False

The first operand is true, and the result will depend on the second operand;

False && true = False

False && false = False

The first operand is false, and the result is thus determined to be false;

Or:

true | | True = True

true | | False = True

The first operand is true, and the result is thus determined to be true;

False | | True = True

False | | False = False

The first operand is false, and the result will depend on the second operand;

Non -:

! true = False

! False = True

(10) Special variables:

$: script file path itself;

$#; The number of script parameters;

$*: All Parameters

[Email protected]: all parameters

(10)

Kernel compilation process:

Steps:

~]# Tar xf linux-3.10.67.tar.xz-c/usr/src extracts its kernel source tarball and then identifies the extracted directory.

~]# CD/USR/SRC then enter this directory

~]# ln-s linux-3.10.67 Linux to make a link to its extracted files, link to Linux

~]# CD Linux into the Linux directory

~]# make menuconfig Configure kernel options

~]# make [-j #] Compile the kernel, you can use-j to specify the number of compiled threads, for example: Make-j 4

~]# make Modules_install installing kernel modules

~]# make install for kernel installation

Linux kernel architecture and underlying--reading notes