[Chinese-English control] User-space Device Drivers in linux:a first look

If you are interested in Linux user-state driver development, please read this article, otherwise please drift over.

user-space Device Drivers in linux:a First Look | initial knowledge of Linux User state device drivers

user-space Device Drivers in linux:a first look Mats liljegrensenior software ArchitectDevice drivers in Linux is traditionally run in kernel space, but can

Drivers in user space, trying to answer the questions in what degree
The driver can run in user space and what can is gained from this?

Linux device drivers typically run in kernel space, but can also run in user space. This article describes the device drivers that run in user space, trying to answer the following two questions: the extent to which the driver is running in user space, and the benefits derived from it.

In the ' 90s, user-space drivers in Linux were much about howto make graphics run faster[1] by avoiding calling the kernel. These drivers where commonly used by the X-windows server. User-space driver have become ever more important, as a blogpost by Tedd Hoff[2] illustrates. In he case the kernel was seen asthe problem when trying to achieve high server connection capacity. Network interface hardware companies like Intel, Texas Instrumentsand Freescale has picked up on this and is now Providi Ngsoftware Solutions for User-space drivers supporting their hardware.

In the 90 's, the user-space drivers in Linux focused on how to make graphics run faster by avoiding kernel calls. These drivers are typically used on x-windows servers. User space drivers are becoming more and more important, as discussed in the blog published by Tedd Hoff. In his case, the kernel is considered to be the problem, when trying to provide high concurrent connection server capability (Note: c10k problem). Network interface hardware companies such as Intel, Texas Instruments, and Freescale have begun to study the problem, and they are now providing software solutions for user-space drivers that support their hardware.

1. Problems with Kernel-space drivers kernel space driver problems

Device drivers normally run in kernel space, since handlinginterrupts and mapping hardware resources require Privilegestha T only the kernel space are allowed to. However, it is notwithout drawbacks.

Device drivers typically run in kernel space because the interrupt processing and hardware resource mappings require privileges, and the corresponding privileges are only allowed in the kernel space. However, it is not without its shortcomings.

1.1 system call overhead overhead

Each of the kernel must perform a switch from user Modeto supervisor mode, and then back again. This takes time, which canbecome a performance bottleneck if the calls is frequent. Furthermore, the overhead is very much non-predictable, whichhas a negative performance impact on real-time applications.

Each call to the kernel must be switched from user mode to Super-admin (kernel) mode before returning. This obviously takes time, and if the call is frequent, it becomes a performance bottleneck. In addition, the overhead is largely unpredictable, which can have a negative performance impact on real-time applications.

1.2 Steep learning curve learning curve steep

The Kernel-space API is different.  For example, malloc () Needsto is replaced by one of the several types of memory allocationsthat the kernel can offer, such As Kmalloc (), Vmalloc (), alloc_pages () or get_zeroed_page (). There is a lot to learn before becomingproductive.

The kernel space API is different than the user space API. For example, if you replace malloc (), the kernel provides several different types of memory allocation APIs, such as Kmalloc (), Vmalloc (), alloc_pages (), or get_zeroed_page (). Want to be effective in the kernel programming, need to learn a lot of things.

1.3 Interface Stability interface stability

The Kernel-space API is less stable than User-space APIs, makingmaintenance a challenge.

Compared to the API of user space, the kernel space API is more unstable, which undoubtedly poses a great challenge to the maintenance of code.

1.4 Harder to debug debugging more difficult

Debugging is very different in kernel space. Some Tools oftenused by User-space applications can is used for the kernel. However, they represent exceptions rather than rule, wherelttng[3] is an example of the exception. To compensate Forthis, the kernel have a lot of debug, tracing and profiling Codethat can is enabled at compile time.

In kernel space, debugging differs and is very different from user-space debugging. Some of the tools that are often used in user space application debugging can be used for kernel debugging. However, they represent anomalies rather than normality, for example Lttng[3] is an exception. To compensate for this, the kernel has many debugging, tracing, and parsing code that can be enabled at compile time.

1.5 Bugs More fatal error more deadly

A crashing or misbehaving kernel tends to has a more severeimpact on the system than a crashing or misbehaving applicatio N,which can affect robustness as well as how easy it was to debug.

A kernel crash or improper behavior affects the system more than an application crashes or is incorrect, which affects the robustness of the system and how easy it is to debug.

1.6 Restrictive Language Choice programming language selection is strictly limited

The kernel space is a very different programming Environmentthan user space. It's more restricted, for example only C Languageis supported. This rules out any script based prototyping.

Kernel space is very different from the programming environment of user space. It is subject to more restrictions, such as only C language support. This excludes any script-based prototyping design.

2. User-space Drivers User space driver

If there, many problems with has device drivers inkernel space, is it time to has all drivers in user space Inst ead? As always, everything have its drawbacks, user-space drivers Areno exception. Most of the issues with kernel-space drivers is solved by havingthe driver in user space, and the issue with interface St Ability Isonly true for very simple user-space drivers. For further user-space drivers, many of the interfacesavailable for kernel-space drivers need to be RE-IMPLEMENTEDFO R user-space drivers. This means, interface stability would stillbe an issue.

Since there are a lot of problems with device drivers in kernel space, should all drivers be placed in user space? As always, any solution has drawbacks, and user-space drivers are no exception. Most of the problems with kernel space drivers can be solved by the user-space driver, but the problem with interface stability is only for those very simple user-space drivers. For more advanced user-space device drivers, many interfaces that are available in kernel space need to be re-implemented for user-space drivers, which means that the stability of the interface is still a problem.

3. Challenges with User-space drivers user space device drivers face challenges

There is a fear in the Linux kernel community that user-spacedrivers be used as a tool to avoid the kernel ' s GPLv2 licens E.this would undermine the idea with free open source softwareideas that GPLv2 have. However, this is outside the scope ofthis paper.

In the Linux kernel community, there is a fear that the user space driver is treated as a tool to avoid kernel GPLv2 licensing. This will undoubtedly undermine the idea of open source software GPLv2 has always advocated. However, this is beyond the scope of this article.

Apart from this there is technical challenges for user-spacedrivers.

In addition, the user space device driver also has many technical challenges.

3.1 Interrupt handling interrupt Handling

Without question, interrupt handling is the biggest challengefor a user-space driver. The function handling an interrupt iscalled in privileged execution mode, often called Supervisormode. User-space drivers has no permission to execute inprivileged execution mode, making it impossible for User-spacedrivers T o Implement an interrupt handler.

There is no doubt that interrupt handling is the biggest challenge faced by user-space device drivers. The interrupt handler function is called in the privileged execution mode (also called the Super Admin mode). User-space drivers are not allowed to execute in privileged mode, which makes it impossible to implement an interrupt handler in a user-space driver.

There is ways to deal with this problem:either you do notuse interrupts, which means so you have to poll INSTEAD.O R has a small kernel-space driver handling only the interrupt. In the latter case can inform the User-space driver of Aninterrupt either by a blocking call, which unblocks Whenan in Terrupt occurs, or using POSIX signal to preempt theuser-space driver.

There are two ways to solve this problem: either do not use interrupts or have a kernel-space driver to handle interrupts specifically. In the previous approach, not using interrupts means that polling must be used. In the latter case, you can notify the user of the space driver by blocking the call, either by opening the blocking call when the interrupt occurs, or by using a POSIX signal to preempt the user space driver.

Polling is beneficial if interrupts be frequent, since there isconsiderable overhead associated with each interrupt, due tothe switch from the user mode to the Supervisor mode and back thatit causes. Each poll attempt on the other hand is usually Onlya check for a value on a specific memory address.

If interrupts occur frequently, polling is beneficial because there is considerable overhead for each interrupt, which is derived from switching from user mode to kernel mode and then returning from kernel mode to user mode. On the other hand, every poll is usually just a check of the value at a specific memory address (so polling is good and can reduce overhead).

When interrupts become scarcer, polling would instead do a lotof work just to determine that there is no work to do. This is Isbad for power saving.

When interruptions become scarce, polling will do a lot of work to determine that there is no work to do, which is not conducive to saving energy consumption.

To get power saving if using User-space drivers with polling,you can change the CPU clock frequency, or the number ofcpu s used, depending on work load. Both Alternatives Willintroduce ramp-up Latency When there was a work load spike.

When the user-space driver uses polling, you can modify the CPU's clock frequency based on the workload, or change the number of CPUs used, if you want to save power. Both methods introduce a sharp delay when workload spikes are encountered.

3.2 DMA Direct Memory access

Many drivers use hardware dedicated to copying Memoryareas managed by the CPU to or from memory areas Managedby Hardware D Evices. Such dedicated hardware is called Directmemory access, or DMA. DMA relieves the CPU of suchmemory copying.

Many drivers use specialized hardware to make memory copies, from CPU-managed memory areas to hardware-managed memory areas, or vice versa. This specialized hardware is called DMA (direct memory access). DMA,CPU can be freed from heavy memory copy work.

There is some restrictions on the memory area used FORDMA. These restrictions is unique for each DMA device. Common restrictions is, only a certain physical memoryrange can is used, and that the physical memory range must Beco Nsecutive.

There are some limitations to the memory area used by DMA. These limits are unique to each DMA device. The usual limitation is that only a certain amount of physical memory can be used, and the range of physical memory must be contiguous.

Allocating memory that can is used for DMA transfers isnon-trivial for user-space drivers. However, since dmamemory can be reused, you have need to allocate a pool ofmemory to being used for DMA transfers at start-up . This meansthat the kernel could help with providing such memory whenthe user-space driver starts R kernelinteractions would be needed.

Allocating memory that can be used for DMA transfers is important for user-space drivers. However, because the memory used for the DMA transfer is reusable, only one memory pool needs to be allocated to be used when the DMA transfer is started. This means that when the user-space driver starts, kernel space can provide such a memory, but after that, no further kernel interaction is required.

3.3 Device interdependencies

Devices is often structured in a hierarchy. For example Theclock might is propagated in a tree-like fashion using differentdividers for different devices and offer th e possibility to poweroff the clock signal to save power.

Xxx

There can is devices acting as a bridge, for example a PCI Hostbridge. Need to setup the bridge in order Tohave access to any device connected on the other side ofthe bridge.

Xxx

In kernel space there is frameworks helping a device Driverprogrammer to solve these problems, but those frameworksare no T available in user space.

Xxx

Since It is usually only the startup and shutdown phases Thataffect Other devices, the device interdependencies can Besolv Ed by a kernel-space driver, while the User-space Drivercan handle the actual operation of the device.

Xxx

3.4 Kernel Services

Network device drivers normally interfaces the kernel Networkstack, just like block device drivers normally interfaces the Kernelfile system Framework.

Xxx

User-space drivers has no direct access to such kernel Services,and must re-implement them.

Xxx

3.5 Client Interface

The kernel have mechanisms for handling multiple clientsaccessing the same resource, and for blocking threads waitingfor EV Ents or data from the device. These mechanisms areavailable using standard interfaces like file descriptors, Sockets,or pipes.

Xxx

To avoid using the kernel, the User-space driver needs to inventits own interface.

Xxx

4. Implementing User-space Drivers | User-state device driver implementation

The picture above shows how a user-space driver might is designed. The application interfaces The user-space part of the driver. The User-space part handles the hardware, but uses it kernel-space part for startup, shutdown, and receiving interrupts.

Xxx

There is several frameworks and software solutions available to help designing a user-space driver.

Xxx

4.1 UIO

There is a framework for the kernel called UIO [5][4] which facilitate writing a kernel-space part of the User-space driver . UIO have mechanisms for providing memory mapped I/O accessible for the user-space part of the driver.

Xxx

The allocated memory regions is presented using a device file, typically Called/dev/uiox, where X is a sequence number F or the device. The user-space part would then open the file and perform mmap () on it. After this, the User-space part had direct access to its device.

Xxx

By reading from the same file being opened for mmap (), the user-space part would block until an interrupt occurs. The content, read would be, the number of interrupts that have occurred. You can use a select () on the opened file to wait for other events as well.

Xxx

For user-space network drivers there is specialized solutions specific for certain hardware.

4.2 DPDK

Data Plane Development Kit, Dpdk[6], is a solution from the Intel for User-space network drivers using Intel (x86) hardware. DPDK defines an execution environment which contains user-space network drivers. This execution environment defines a thread for each CPU, called Lcore in DPDK. For the maximum throughput you should is not the other thread running on the CPU.

Xxx

While the libraries focuses on forwarding applications, you can implement server applications as well. For server DPDK applications your need to implement your own network stack and accept a DPDK specific interface for Accessi ng the network.

Xxx

Xxx

4.3 USDPAA

User-space Data Plane Acceleration Architecture, usdpaa[7], is a solution from Freescale for the same use case as DPDK bu T designed for their QorIQ architecture (PowerPC and ARM. The big difference is this QorIQ uses hardware for allocating, de-allocating and queuing network packet buffers. This makes memory management easier for the application.

Xxx

4.4 Transportnetlib

TRANSPORTNETLIB[8] is a solution from Texas Instruments. It is similar to USDPAA and for the Keystone Architecture (ARM).

Xxx

4.5 Open Dataplane

Open Dataplane, Odp[9], is a solution initiated by Linaro to does the same as DPDK, USDPAA and Transportnetlib, but with Ven Dor generic interfaces.

4.6 Trying out DPDK

To get the feeling for the potential performance gain from have a user mode network device driver, a DPDK benchmark APPL Ication was designed and executed.

Xxx

The design of the application can is seen in the picture above. It executes as four instances each running in its own CPU, or Lcore, as DPDK calls them.

Xxx

Each instance are dedicated to its own Ethernet device sending and receiving network packets. The packets sent have a magic word used for validating the packets and a timestamp used for measuring transport latency.

Xxx

The instances is then paired using loopback cables.  To being able to compare User-space driver with Kernel-space driver, one pair accesses the hardware directly using the driver Available in DPDK, and the other pair uses the pcap[13] interface. All four Ethernet devices is on the same PCI network card.

Xxxx

There is a fifth lcore (not shown in the picture above) which periodically collects statistics and displays it to the Scre En.

Xxx

The hardware used was as follows:   o supermicro a1sai-2750f Mother board using Intel Atom     C2750 CPU.  This CPU has 8 cores with no hyperthreading.   o 16GB of memory.   o Intel Ethernet Server Adapter I350-T4, Mbps.

Xxx

The table below shows the throughput and latency for user-space driver compared to kernel-space driver.

Xxx

A graph showing the throughput:

Xxx

A graph showing the latency:

Xxx

The theoretical throughput maximum is the sum of the "send" and receives speed for the network interface. In this case this is the $ Mbps in each direction, giving a theoretical maximum of Mbps. The throughput includes packet headers and padding.

Xxx

User-space driver achieved a throughput boost of about four times over Kernel-space driver.

Xxx

Latency was calculated by comparing the timestamp value found in the network packet with the current clock when packet was Received. The latency for User-space driver is slightly less than for kernel-space driver.

Xxx

Four threads, each continuously running Netperf TCP streaming test against Loop-back interface, were used as a stress whil E running the DPDK benchmark application. This had no noticeable impact on the measurements.

Xxx

5. Conclusion | Concluding remarks

Implementing a User-space driver requires some work and knowledge. The major challenges is interrupts versus polling, power management and designing interface towards driver clients.

Xxx

Support for User-space network drivers are a lot more developed than for other kinds of user-space drivers, especially for Doing data plane forwarding type of applications.

Xxxx

A user-space driver can do everything a kernel-space driver can, except for implementing an interrupt handler.

Xxx

Comparing a user-space network driver with a Kernel-space network driver showed on four times better throughput for the User space driver. Latency did not show a significant difference.

Xxx

The real-time characteristics should is good for user-space drivers since they does not invoke the kernel. This is not verified in this paper, though.

Xxx

[Chinese-English control] User-space Device Drivers in linux:a first look