Linux Network Card Driver Analysis (ne2000 as an example)

1
LinuxNetwork Card Driver: tracing and Performance Analysis
Field of contribution: Network Software
Chen Yiwei Lin yingda
Department of Information Science, National Jiaotong University
No. 1001, Xinzhu University Road
Tel :( 03) 5712121 Ext. 56667
E-MAIL, iwchen@cis.nctu.edu.tw
Main Contact Person: Chen Yiwei Tel: 0927308032
Summary
The network components of a computer can be divided into two parts: hardware and software. In terms of hardware, there are network cards, network routes, etc.
Adapter Driver, protocol driver, and application software.
The program is responsible for linking hardware and software. This article describes the operation process of the device driver program and finds out the key points of writing the device driver program.
Essentials, LinuxThe network card driver is used as an example to describe and test and analyze it.
Under the condition of propagation time, the order of time spent in packet processing is as follows: (1) sent by the Network Adapter
(Transmit), accept (receive) packets, (2) DMA (direct access to memory) packets, (3) Network Card Driver (CPU execution
Time), the proportion of time spent by the three is about 250: 30: 1, which is based on the packet size of bytes.
The larger the ratio of the three, the longer the transmission (or receipt) and DMA packets are the time bottlenecks.
During interrupt processing, it takes a little longer to "New packets arrive" than "packet transfer completed.
Keywords: Linux, Protocol driver, network card driver, ISR, IRQ, I/O port, DMA.
1. Introduction
2
One of the main functions of the operating system is to control the input and output of computer peripheral devices.
For the four parts, see Figure 1. We can see that the driver is in a similar
When it is important, it must be "soft" or "hard". Usually interrupt processing can be regarded as a part of the driver.
Minute.
For Adapter driver, the "soft" that it wants is the upper-layer protocol driver.
Protocol driver is also the TCP/IP protocol stack, and the "hard" it wants is the underlying network card,
In more details, we need to communicate with the Mac controller on the network card through the registers on the Mac controller.
To exchange messages, to achieve one of the main tasks of the Network Card Driver: send packets to the network card or capture from the network card
Another major task is to pass packets captured from the network card to the upper-layer protocol driver,
Or pass the packets received from the protocol driver to the network card. See figure 2.
User itinerary
Device-independent software
Driver
Interrupt handling regular
Hardware
I/O call, spooling
Naming, protection, allocation
Setup device registers, check status
Wakeup driver when I/O completed
Perform I/O operations
I/O request I/O reply
Figure 1: hierarchy in the I/O system and the main function of each layer
I/O function
Core network card driver network card
Exit Packet
Inbound packets
Exit Packet
Inbound packets
Figure 2: main work of the network card driver
3
2. Write the driver
How to properly LinuxWrite a driver in the operating system. We will divide it into probe hardware (
Hardware), interrupt processing (interrupt handling), read and write data using the I/O port (using I/O ports ).
Most of them are explained one by one.
2.1 probe hardware)
Before a driver can communicate with the device, some initialization work that must be done includes: the I/O of the probe Device
Ports and IRQ Number. with I/O Ports, you can communicate with some registers on the device. With IRQ
Number can be correctly registered to the core as the interrupt handling regular. methods for detecting hardware and types of Bus
The PCI interface device automatically stores the I/O Ports and IRQ Number when the boot is started.
But the driver only needs to read these latches, and does not need a real probe device,
However, the device driver of the ISA interface must actually perform the probe.
In terms of the PCI interface, LinuxCore version 2.4 further integrates I/O Ports and IRQ Number in system resources
Source (pci_dev structure), so in the driver, it is not necessary to directly obtain the I/O ports from the device
And IRQ Number, which can be obtained by system resources by calling the following function:
Int pci_enable_device (struct pci_dev * Dev );
Unsigned long pci_resource_start (struct pci_dev * Dev, int bar );
Struct resource * request_region (unsigned long start, unsigned long Len, char *
Name );
Void release_region (unsigned long start, unsigned long Len );
Before obtaining the I/O Ports and IRQ Number, call pci_enable_device () to start
If IRQ Number is required for this PCI device, it can be obtained from Dev-> IRQ.
I/O Ports calls pci_resource_start () to obtain the base address, and then calls
Request_region () can be used to declare a certain segment of I/O ports to the system.
Call release_region () to release the I/O Ports.
As a result, the setup of the ISA interface is much more troublesome. The mechanism used in the detection of I/O ports is
"Scan" (SCAN) all possible I/O Ports. Only the I/O ports connected to the device will respond correctly,
4
The following is a rough process:
1. Int check_region (unsigned long start, unsigned long Len );
This function is used to check whether I/O ports can be used.
2. Check whether the device exists and try to avoid "writing" because you are not careful.
Write to the system will cause many problems.
3. Call request_region () to request the I/O ports required by the core.
4. When I/O ports are not used, call release_region () to release system resources.
The mechanism used to detect the IRQ Number is to interrupt the device (Interrupt). However
Later, check the relevant information to find out how much IRQ the device uses.
The program needs to know the number of IRQ used by the device, rather than assigning IRQ to the device. The following is
Functions:
Unsigned long probe_irq_on (void );
Int probe_irq_off (unsigned long );
Probe_irq_on () returns a bitmask, which can be used to determine which other
The driver must store this mask because the mask must be used as a parameter.
Data transmission to probe_irq_off (). After probe_irq_on () is executed, the driver should prompt the device
Call probe_irq_off (bitmask) after an interruption occurs, so that probe_irq_off ()
The IRQ Number used by the device is returned.
2.2 interrupt handling (interrupt handling)
When data is transmitted between the core and the device, there will inevitably be some latency (Delay), so the driver must
Store the data in the buffer zone and wait for the complete and correct data transfer time
And our common buffer mechanism is "interrupt-driver I/O ".
The input buffer is a program that is used to fill in the data when the processing is interrupted.
(Process); output buffer is filled with data by a program, where
5
And then send the data to the device. The so-called interruption processing part is handed over to the interruption processing Regular Expression for negative
Responsibility. In general, a driver will only register an interrupt handling Regular Expression for its device to the core,
The following functions can be used for registration and cancellation interrupt handling:
Int request_irq (unsigned int IRQ, void (* Handler) (INT, void *, struct pt_regs *),
Unsigned long flags, const char * dev_name, void * dev_id );
Void free_irq (unsigned int IRQ, void * dev_id );
Request_irq can be used to register the interrupt handling regular expression, while free_irq is used to cancel it.
2.2.1 interrupt handling outline
When an interruption occurs, the system is "both soft and hard". See table 1.
1. hardware storage program counters (programm conuter), etc.
2. Load hardware into a new program counter
3. The group language program stores the value of the temporary storage device.
4. Set a new stack for the group Language Program
5. C program handles actual interrupted transactions, wakes up the itinerary, and may call
Schedule (), and finally jump back to the group Language Program
6. The group language program starts the current execution schedule
The entire process of interrupt service routine (ISR) is 3 ~ 6, and the driver is
5 items.
2.2.2 "quick" and "slow" handling regular
In the old version LinuxIn the core, it takes a lot of time to distinguish between "fast" and "slow" interruptions.
The difference between "fast" and "slow" interruptions lies in the length of processing time required.
A short period of time can be processed, that is, a fast interruption. On the contrary, if an interruption takes a long time
To complete the process is a slow interrupt. Compared with the Interrupt Processing formula, when the interrupt is processed, the processor
Whether it will accept other interrupt return (Interrupt reporting). If yes, the interrupt is often handled.
Table 1: outline of interrupt handling
6
Type is slow (because the processor must take into account other interruptions), and vice versa (because the processor
It is completely occupied by the interrupt handling regular). The two have the same place, either fast or
The interrupt disposal type of slow, when it is in the process of a certain interruption service, this interrupt is in the Interrupt Controller
(Interrupt Controller) is invalid (disable). Flags parameter in request_irq ()
If it is set to sa_interrupt, it indicates that you want to register a fast interrupt handling regular expression.
In the core of generation, fast and slow interruptions have been considered almost the same. Table 2 shows the two
Comparison:
During interruption
It breaks down and becomes invalid.
Make the current
Service Interruption failure
Will the call be made after the service ends?
Call
Ret_from_sys_call
Fast processing Regular Expression Yes No
Slow processing regular expression no Yes
2.2.3 write interrupt handling regular
The work that must be taken into account when preparing the interrupt handling formula is listed in table 4.
Last entry in Table 4
We will introduce the subsequent regular expression in the next section.
Interrupt handling is typically performed during the interruption period, so the execution is subject to many restrictions,
Identify the cause of the interruption. e.g. It is usually necessary to determine that the interruption is caused by packets.
The arrival, departure, or error.
Wake up the itinerary of the event waiting for completion of the interruption.
If a long execution time is required, use the bottom half.
Table 2: Comparison between fast and slow interrupt handling
Table 4: routine handling of interruptions
7
For example, it cannot communicate with the user space, and it cannot do anything that will make itself sleep.
The resources that we can use in the process of implementing Interrupt Processing
The three parameters are int IRQ, void * dev_id, and struct pt_regs * regs.
IRQ can be used in the recording file. dev_id is the indicator of this device. When we use shared interrupt (e.g.
Two Interrupt Processing Regular Expressions share One IRQ Number), and the Interrupt Processing regular expression can use dev_id
Identify whether an interrupt belongs to the one you want to handle, and the last parameter regs is rarely used,
The regs stores the content of the processor before it enters the Interrupt Processing formula, so the regs can
Used for monitoring and debugging.
2.2.4 subsequent processing regular (bottom half)
One of the main problems encountered when handling interruptions is how to make the handling routine run smoothly
For time-consuming work, the next interruption will not be missed, LinuxSolution
The interrupt handling formula is divided into two parts: top half and
One part is "bottom half". The first feature is the one we mentioned before.
The disposal regular expression registered by request_irq, which schedules the subsequent regular expression to the operating system
Security time that can be implemented by the Business System (the so-called security means that the execution time is not strictly required
Continue. LinuxThere are two mechanisms in the core to implement the subsequent regular expression: BH (
Also known as bottom half) and tasklets. bh are older methods. In fact, after core 2.4
It also uses tasklets for implementation, and tasklets was developed from the 2.3 series.
This method is often used for subsequent execution. However, it cannot be used by tasklets on older cores.
So if you consider portability, it is more appropriate to use the BH method.
To implement the subsequent regular expression using the tasklets method, which functions can be used:
Declare_tasklet (name, function, data );
Tasklet_schedule (struct tasklet_struct * t );
For example, if you write a function func (),
The first step is to declare a tasklet to the system: declare_tasklet (task,
8
Func, 0), task is the name of this tasklet, and then call tasklet_schedule (& task)
To schedule the task to the job system. When the job system is convenient, it will be executed.
In the BH method, if you want to schedule a subsequent constant to the system, you can use the following function:
Void mark_bh (INT nr );
Specifically, NR represents a bottom-half routine. In the old version of the core, mark_bh () will
Bitmask sets a certain dollar, so that the Interrupt Processing Regular Expression corresponding to this dollar can be smoothly
When executed, the core usually provides several subsequent constants for program designers to use. Table 3 lists the drivers.
What programs can use:
However, in core (2.4) of the new version, mark_bh () actually calls tasklet_hi_schedule ()
(Like tasklet_schedule () to shift the bottom-half routine to the core.
2.2.5 Race Condition)
Due to interrupt-driver I/O, a certain type of problem is introduced:
How can we solve the synchronization problem of the shared data that needs to be accessed during several trips?
It is called race condition. As a result, it may occur at any execution point of the driver.
Interrupt, so as long as the driver interacts with the interrupt (most of the drivers will), you must pay attention to the competition.
LinuxThe core supports several technologies to avoid data corruption.
Method of use: Use spinlocks to achieve mutual exclusion (Mutual Exclusion) between the travel, so as to avoid competition
The occurrence of contention State indicates that the white point is whether the Interrupt Processing regular type or the driver needs to be stored
When retrieving shared data, you must first get a lock before accessing the shared data.
Return the lock (unlock) so that other trips can access the shared data in the same way.
In LinuxThe type in is spinlock_t, and the following are some functions that can operate on the spinlocks:
Immediate_bh tqueue_bh timer_bh
Table 3: Subsequent continuation of core announcements
9
Void spin_lock (spinlock_t * Lock );
Void spin_lock_irqsave (spinlock_t * Lock, unsigned long flags );
Void spin_lock_irq (spinlock_t * Lock );
Void spin_lock_bh (spinlock_t * Lock );
Void spin_unlock (spinlock_t * Lock );
Void spin_unlock_irqrestore (spinlock_t * Lock, unsigned long flags );
Void spin_unlock_irq (spinlock_t * Lock );
Void spin_unlock_bh (spinlock_t * Lock );
Spin_lock () gets the desired lock in the form of busy-wait. Wait
When the spin_lock () function returns, you can obtain the lock by calling its itinerary.
Spin_lock_irqsave () also acquires a lock in the same way, which causes interruption.
Invalid, and records the current interruption to the flags parameter, while
Except that the current interruption is not recorded, the remaining actions are the same as those of the spin_lock_irqsave () operation.
Spin_lock_bh (), which is the same as the first three, uses the busy-Wait method to obtain a lock. However, it also
The subsequent regular expression is not allowed to be executed. The above four operators can obtain a lock. On the contrary, the following four correspond
The function will return the lock, and the spin_unlock () will simply return the previous lock,
Spin_unlock_irqrestore () will return the lock and make some interruptions effective based on the flags value,
In addition to returning the lock, spin_unlock_irq () also makes all the interruptions valid, while
Return the lock and allow the subsequent regular expression to be executed. Pay special attention to the following when using these functions. The first is
Indeed
The function used to obtain the lock is earlier than the function used to return the lock. The second is to pair these functions in pairs.
.
2.3 read and write data using I/O Ports
After the hardware phase is tested, the driver can obtain the I/O ports used by the hardware. Most
Hardware divides the width of I/O ports into 8 bits, 16 bits, and 32 bits.
10
To access the I/O Ports of different widths. Linux
The core defines the following functions for accessing I/O Ports.
Unsigned INB (unsigned port );
Void outb (unsigned char bye, unsigned port );
INB () reads ports with a width of 8 bits, while outb () writes ports with a width of 8 bits.
Unsigned inw (unsigned port );
Void outw (unsigned char bye, unsigned port );
Inw () reads ports with a width of 16 bits, while outw () writes ports with a width of 16 bits.
Unsigned INL (unsigned port );
Void outl (unsigned char bye, unsigned port );
INL () reads ports with a width of 32 bits, while outl () writes ports with a width of 32 bits.
In addition to the preceding single read method, LinuxString operations are also supported:
Void InSb (unsigned port, void * ADDR, unsigned Long Count );
Void outsb (unsigned port, void * ADDR, unsigned Long Count );
InSb () reads the data of the count-bit tuples from the 8-bit ports, and then stores the data to the memory.
Where the body address is ADDR, and outsb () writes the data whose memory address is ADDR to the Count bit
Tuples to 8-bit ports.
Void insw (unsigned port, void * ADDR, unsigned Long Count );
Void Outsw(Unsigned port, void * ADDR, unsigned Long Count );
The operations of these two functions are similar to those of the above two, except that they are for a width of 16 bits.
Ports.
Void insl (unsigned port, void * ADDR, unsigned Long Count );
Void outsl (unsigned port, void * ADDR, unsigned Long Count );
The operations of these two functions are similar to those of the above two, except that they are for 32-Bit Width
Ports.
11
3. LinuxNetwork Card Driver instance
Among the home network card drivers, we select the ne2000 with the highest compatibility and the PCI interface network.
Road card LinuxVersion of the driver for tracking, so more practical application significance.
3.1 important data structure)
In LinuxIn the system, there are two types of data structures commonly used by the Network Card Driver: sk_buff and
Net_device. Figure 4 shows the locations of the two data structures in the system.
Who will access them.
Simple Description 4: In the figure, SKB refers to the sk_buff type indicator variable, and Dev refers
Indicator variable in the net_device form. The NIC Driver captures packets from the NIC (at this time
Is called frame), will configure a sk_buff memory, frame into the data in SKB
The field in the system is represented as sk_buff.
The multi-column value is filled in the NIC Driver. In the figure, the local value refers to most of
The field values are generated in the NIC driver and then filled in the net_device field.
In the system, it is represented in the form of net_device. Next we will introduce these two documents in detail.
The main column.
Core
SKB
Dev
Network Card Driver
SKB Frame
Dev local
Network Card
Figure 4: Position of skb_buff and net_device IN THE SYSTEM
12
3.1.1 sk_buff
The data structure is defined in table 5, which lists its main columns and columns.
Meaning.
3.1.2 net_device
The data structure is defined in Table 6, which lists its main columns and
The meaning of the column.
3.2 Initialization
Initialization can be divided into "registration interrupt handling regular" and "detection Hardware", registration interrupt
Processing is often done using the previously mentioned request_irq () function, and the hardware aspect of detection is
The network card of the PCI interface, so we only need to use the methods we have taught to obtain the I/O Ports and IRQ
Number, instead of actually detecting the network card.
Column name meaning
Name device name
Base_addr I/O port base address
IRQ interrupt number
Dev_addr network card hardware address
Maximum Transmission unit of MTU
One of the functions of the hard_start_xmit Device
Column name meaning column name meaning
Head points to the start point of sk_buff. The length of Len data itself.
Data points to the start point of "real data" pkt_type helper package type
Tail points to the end of "real data" saddr Source Address
End Point to the end of sk_buff daddr Destination Address
Raddr router address of the device where the dev package arrives or leaves
Table 5: key columns of sk_buff
Table 6: Important columns of net_device
13
3.3 packet transmission
Figure 5 shows the packet sending process of the NIC Driver. The shadow part is the driver.
Here, we will briefly describe figure 5: the core is to send packets and call Dev-> hard_start_xmit (),
The function implemented in this driver is ei_start_xmit (). In ei_start_xmit (), it calls
Ne2k_pci_block_output (): Move the packets to the network adapter, and then call
Ns8390_trigger_send () to trigger the network adapter to send packets. After the packets are sent,
The network card will notify the core of the interrupt, so the core will call the corresponding interrupt handling regular expression.
In a program, it is ei_interrupt (). ei_interrupt () first determines which interrupt is the cause,
After the packet is transmitted, it will call ei_tx_intr () for processing. ei_tx_intr ()
Call ns8390_trigger_send () to trigger another packet transfer on the network card.
Then, call netif_wake_queue () to let the core know that the packet has been passed.
Work.
Core
(Conventional disposal)
Ei_interrupt
(TX)
Ei_start_xmit
(RX)
Ei_receive
Ei_tx_intr
Network Card
1. Dev-> hard_start_xmit
2. ne2k_pci_block_output
3. ns8390_trigger_send
4. Interrupt generation
5.
6.
7. ns8390_trigger_send
8. netif_wake_queue
Figure 5: Packet sending Process
14
3.4 receive packets
Figure 6 shows the packet receiving process of the NIC Driver.
Figure 6: when the network card receives the packet, it will notify the core of the interruption.
Is the interrupt handling Regular Expression corresponding to the core call, in which the driver is ei_interrupt (),
Ei_interrupt () will first determine which type of interruption it belongs to, and discover that a packet is sent as a result.
After it is disconnected, it will call ei_receive () for processing, and will call in ei_receive ()
Ne2k_pci_block_input () transfers packets from the network card to the system memory, and then inserts the packets
In the sk_buff structure and in the form of netif_rx (), this packet is dropped to the upper layer for processing, while netif_rx ()
What we have done is what we have discussed before and what we should do in the continued regular expression.
4. Performance Analysis
Here we will do two interesting performance analyses, one named "the lifetime of the packet" and the other named "the duration of the interrupt processing ".
". The first test shows us how long an exit and entry package stays in all parts of the system.
You can know where packet processing takes time. The second test will let us know that the Network
The duration of handling different interruptions of the card driver is completely different, and the categories of interruptions are different (transfer completed, receive
There are different processing methods for new packets. We can use this test result to determine the network card and drive card drivers.
One of the factors is whether or not the combination is good, because the better the combination of the network card and the network card driver, the average time spent on Interrupt Processing
Core
(Conventional disposal)
Ei_interrupt
(TX)
Ei_start_xmit
(RX)
Ei_receive
Ei_tx_intr
Network Card
1. interrupt occurs
2. 3.
4. ne2k_pci_block_input 5. netif_rx
Figure 6: Packet acceptance process
15
The shorter the period.
The lifetime of 4.1 packets
We divide a package into three phases: the network card phase, the DMA phase, and the network card drive.
The dynamic program phase (excluding the DMA part) will measure how long the Exit and Entry packets are waiting in these three phases.
The measurement method is to first determine the start point of each stage and then use rdtscll (defined in) to capture
Take the time. Table 7 shows the hardware used for this test. Software:
CPU: celon 567.007 MHz
Network Card: PCI RealTek 10 Mbps Ethernet Card
Core operating system: LinuxKernel 2.2.19
Network Card Driver: ne2k-pci
The following table 8 shows the result of this test. The unit of time is the number of CPU clock cycle,
Since our CPU speed is 567.007 MHz, a clock = 1.8 nanosecond.
In the network card phase, the time of entry and exit packets is proportional to the package size.
Package category size (byte) network card phase DMA phase Network Card Driver phase
ICMP (28 + 1)
ICMP (28 + 100)
ICMP (28 + 1000)
Entry and exit
62000 12000
102000 52000
512000 462000
Entry and exit
7000
11000
66000
Entry and exit
1300 3000
1300 3000
1300 3000
UDP (28 + 1)
UDP (28 + 100)
UDP (28 + 1000)
62000 12000
102000 52000
512000 462000
7000
11000
66000
1300 3000
1300 3000
1300 3000
ARP (28) 62000 12000 7000 1300 3000
Table 7: Test 1 software and hardware devices
16
The time it takes for the network to be stuck in the transfer packet to the network, and to receive packets from the network is
The reason why the incoming packets are longer than the outgoing packets in the network card phase is:
Incoming packets: receipt time + combined language Interrupt Processing Time + C language interrupt processing time.
Outbound packet: Transfer Time + processing time of some C language interruptions.
Because the entry package takes more time to interrupt the processing of the combined language, coupled with its C language Interrupt Processing
The entry package remains on the network card for a longer time than the exit package.
In the phase, the waiting time of the packet is obviously only related to the packet size, whether it is to enter or to exit
And the minimum waiting time for packets in the Network Card Driver phase, especially for inbound traffic.
Packet, this is because when the incoming packet DMA is connected to the NIC driver, the NIC driver will establish
In the instant call, netif_rx will hand over the packet to the subsequent regular expression for processing.
4.2 Interrupt Processing Time
In this test, we found two network cards and their corresponding network card drivers to test the two
Of course, this is also the time that drivers spend on interrupt processing.
Packet comes to these two types of interruptions, table 9 is the software and hardware used for this test.
CPU: celon 567.007 MHz
Network Card: PCI RealTek RTL-8029 ()
10 Mbps Ethernet Card
Network Card Driver: ne2k-pci
Network Card: PCI intel gd82559 100 Mbps
Ethernet Adapter
NIC Driver: eepro100
Core operating system: LinuxKernel 2.2.19
Table 9: Test 2 software and hardware devices
17
Table 10 is the result of this test. The unit of time is the number of CPU clock.
Interrupt type
Packet size (byte)
New package comes
Eepro100 ne2k-pci
Packet transfer completed
Eepro100 ne2k-pci
28 + 1 8500 29000 6000 8500
28 + 100 8500 34000 6000
28 + 1000 8500 90000 6000
The result shows the average duration of the combination of eepro100 and intel network card in terms of Interrupt Processing.
Compared with the combination of ne2k-pci and RealTek network card, it can be said that eepro100 and
The combination of the intel network card is better than the combination of the ne2k-pci and RealTek network card; in addition, we
It can be found that the time required for "new packet arrival" is much longer than "packet transfer completed" because
When processing a new package, you must first extract the package from the system memory on the network card DMA, and then plug it in.
Sk_buff, and transfer it to the upper layer, and when processing "packet transfer completed", it is only
For example, adding the number of transmitted packets to a level 1.
5. Conclusion
The network card driver plays an important role in the system, because it must communicate with the lower-layer network card and the upper-layer core.
Everything you have to do is very diverse and complex. If you want to learn how to write the NIC driver, the best way is to find an example.
The method described in section 2 is quite generic, not just for the network card driver, but for other device drivers.
In section 4, we have made two interesting tests. From these tests, we can learn more about packet transmission.
The relationship between the delivery and the network card driver, from the "packet life" test, you can know that when processing packets, DMA and send (or receive
Packets are a time bottleneck, so you may design a way to speed up processing packets.
6. References
[1] Alessandro Rubini ," LinuxDevice Drivers ", first edition, O 'Reilly & Associates,
1998.
Table 10: Test Result 2
18
[2] Alessandro Rubini & Jonathan Corbet ," LinuxDevice Drivers ", 2nd edition, o'reilly
& Associates, June 2001.
[3] Andrew S. Tanenbaum, "Modern Operating Systems", Prentice Hall, 1996
[4] W. Richard Steven S, "TCP/IP replicated strated, volume1 & 2", Addison Wesley, 1994
[5] Lin yingda, "computer network experiment", Vico Press, 1999.