ULP (Upper Layer Protocol) brought in by the Linux 4.13/4.14 kernel

Order

After a very cool national Day holiday, with a small little boyfriend parents back to their hometown tasted Chaoshan food, Nanao Island fishing shrimp, seafood support to explode, back to Shenzhen the next day little other children parents took us to Dongguan Changan tasted the authentic Enshi soil home vegetables, a few days down to drink a few good wine, There is no better place to eat and drink, thanks to the small expansion of our social circle ...

?? I think of 14 in Shanghai with a neighbor friend from Jiading to Suzhou, and then in a Jinji lake on the side of the Japanese store made to emerges and not homesickness, pondering the total dripping jiangnan rain in the feeling, think of those a few years ago those Xu a drunken fang after the branch dendrite Vine, In addition to the holiday to see rock June public number about the Tang Dynasty band Ding Wu Post, more is boiling blood, and then Wenzhou leather shoes factory boss went to Cxs Qinghai-Tibet Plateau travel Let me also recalled eight months ago that unforgettable trip to the plateau (but Wenzhou boss is going into Dian line), sinus only big fairy again out of the new song and the theme song of Creation) and controversial ... Nine taste mixed many, I can only silently see and think, this is not, precipitate down is about the Linux 4.13/4.14 kernel ulp Some content, deliberately summed up some disorderly language, to show their living reality.

Yet another Ktls

The Linux 4.13/4.14 kernel brings a new gadget called ULP (Upper Layer Protocol), which is a framework that was introduced to support KTLS (Kernel TLS supports), which is to support TCP BBR and refactoring the entire TCP implementation. ULP is completely tailor-made for ktls, but only extra, it's just right to do something else very beautiful. A Dmeo is given later in this article.

?? What the? Ktls? Isn't that a long time ago?

?? Yes, the idea of supporting TLS in the Linux kernel has already been put forward by Facebook, and I've written about this before:
KTLS (Kernel SSL/TLS) principles and examples from Facebook: http://blog.csdn.net/dog250/article/details/53868519
So what is the difference between ktls in the Linux 4.14 kernel and the original KTLS implementation of Dave Watson? The answer is that the 4.14 Ktls uses ULP as its underlying support, the beauty of which is that ULP not only supports KTLS, but also serves as a universal framework to support other functions.

?? And for the original KTLS, it is based on a new type of socket to support the TLS logging protocol, this implementation is not elegant, if you want to implement another protocol, you need to come up with a socket type, will eventually create a nightmare like Unix sockopt. ULP, it is simply a "any upper layer" tiled above the "transport", which perfectly maps the 7-layer model of the OSI model!

Linux protocol stack

From the protocol layering model, Linux only implements the TCP/IP model, so in the Linux kernel stack is not see the layer above the transport layer, if you want to implement a presentation layer or session layer, you must use some kind of user-State Library, such as the SSL/TLS handshake protocol as the session Layer protocol , and the SSL/TLS logging protocol as the presentation layer protocol, you can use OpenSSL to implement (Libssl,libcrypto), but they are not supported in the kernel stack.

?? On the other hand, from the BSD socket interface, the socket provides only one interface specification, and does not specify how the interface is implemented. We know that in a Windows system, the socket program must initialize the SPI, that is, it must start with WSAStartup and install the Winsock of the specific vendor's socket implementation, The behavior of the socket can be tied to the implementation of a particular Winsock service provider, and I wrote this demo a long while ago:
How OpenVPN on Windows encapsulates the real IP as the source address: http://blog.csdn.net/dog250/article/details/7988939
When I played the TAP network card earlier, Linux can now implement similar functions.

Motivation

If you want to ask some of the coding, encapsulation and other functions in the kernel to achieve what is the benefit, the answer is not only this can improve performance so simple, many people treat the user state implementation and the kernel implementation of the comparison between the total likes to start from performance, which is far from correct. The kernel supports a lower level of support than the library, allowing you to stay away from the compatibility caused by library dependencies, and allows you to encapsulate data with new protocols without changing a line of user-state code.
The temptation is big enough! In fact, many times performance is not a concern, the user-State Library implementation performance is now comparable or even beyond the core implementation, as for the user state and the kernel state of the data copy overhead, there are many different ways to solve.

Linux 4.14 Ktls

The Linux 4.14 kernel has built-in support for KTLS, but if you use it, it is important to note that the current KTLS is not a complete TLS implementation, it only implements a portion of the Logging protocol (symmetric encryption, which is still not implemented by symmetric decryption operations) and does not implement the handshake protocol at all. The relevant data links are listed below:
KTLS Support-ulp:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=734942cc4ea6478eed125af258da1bdbb4afe578
ULP Description:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=99c195fb4eea405160ade58f74f62aed19b1822c
KTLS implementation:
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?www.zzktv.cn id= 3c4d7559159bfe1e3b94df3a657b2cda3a34e218
Looking closely at this implementation, it will be found that although it does not fully implement the entire TLS, but built a good shelf, more than the original version of Dave Watson is much more structured, on the basis of ULP, all the functions can be easily implemented, so that the implementation of more attention to business logic rather than platform-related details , this is how OO Ah!

UTP Demo-caesar Cipher

Well, here's what I want to focus on in this article, which is the text.

?? In this article, I want to give a demo to achieve a seamless Caesar encryption transmission, so-called seamless, that is, the interaction between the two sides do not know their own data is encrypted by the Caesar encryption algorithm, the application still use SEND/RECV such socket interface, However, only through an explicit setsockopt or a hijacked setsockopt call, the data is encrypted, when the data arrives at the receiving end, it is naturally decrypted and restored. I'll uncover the secret!

?? First look at the transmission of the two sides of the code, I use a typical TCP-based C/s program for example, first I give C and s code:

Service-side code (compiled into server)
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/socket.h>
#include <netinet/in.h>
#include <arpa/inet.h>
#include <netdb.h>
#include <errno.h>
#include <sys/www.rbuluoyl.c types.h>
#include <netinet/tcp.h>
#include <unistd.h>

int port = 1234;
The keys on both sides must be identical
int key = 12354;

#define TCP_ULP 31
#define Tcp_none 100

int main ()
{
int sockfd;
int ret;
Char buf[256];
struct sockaddr_in serveraddr;
struct sockaddr_in clientaddr;
int addr_len = sizeof (CLIENTADDR);
int clientfd;

Bzero (&serveraddr, sizeof (SERVERADDR));
serveraddr.sin_family = af_inet;
Serveraddr.sin_port = htons (port);
SERVERADDR.SIN_ADDR.S_ADDR = htonl (Inaddr_any);

SOCKFD = socket (af_inet, sock_stream, 0);
if (SOCKFD = =-1) {
Perror ("Socket error_1");
return 1;
}

ret = bind (SOCKFD, (struct sockaddr *) &serveraddr, sizeof (SERVERADDR));
if (ret = =-1) {
Perror ("Bind error_1");
Close (SOCKFD);
return 1;
}

RET = Listen (SOCKFD, 5);
if (Ret < 0) {
Perror ("Listen");
Close (SOCKFD);
return 1;
}

while (1) {
CLIENTFD = Accept (SOCKFD, (struct sockaddr*) &clientaddr, (socklen_t*) &addr_len);
if (Clientfd < 0) {
Perror ("accept");
Continue
}
Set up to use ULP that supports Caesar encryption
SetSockOpt (CLIENTFD, sol_tcp, Tcp_ulp, "Caesar cipher", sizeof ("Caesar cipher"));
Perror ("ULP init");
Set the key for the encryption algorithm
SetSockOpt (CLIENTFD, Sol_tcp, Tcp_none, &key, sizeof (int));
Perror ("cipher init");

ret = recv (clientfd, Www.gouyifl.cn/buf, sizeof (BUF), 0);
if (Ret < 0) {
Perror ("recv error");
Close (CLIENTFD);
Close (SOCKFD);
return 1;
}
Buf[ret] = 0;
printf ("%s\n", buf);
Close (CLIENTFD);
}
Close (SOCKFD);
return 0;
Client code (compiled into client)
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/socket.h>
#include <netinet/www.thd580.com in.h>
#include <arpa/inet.h>
#include <netdb.h>
#include <errno.h>
#include <sys/types.h>
#include <netinet/tcp.h>
#include <unistd.h>

int port = 1234;
The keys on both sides must be identical
int key = 12354;

#define TCP_ULP 31
#define Tcp_none 100

int main ()
{
int sockfd;
int i = 0;
int ret;
struct sockaddr_in serveraddr;
Char *buf = "ABCDEFG";

Bzero (&serveraddr, sizeof (SERVERADDR));
serveraddr.sin_family = af_inet;
Serveraddr.sin_port = htons (port);
SERVERADDR.SIN_ADDR.S_ADDR = inet_addr ("127.0.0.1");

SOCKFD = socket (AF_www.tianhengyl1.com INET, sock_stream, 0);
if (SOCKFD = =-1) {
Perror ("Socket error!");
return 1;
}
Set the ULP with Caesar encryption
SetSockOpt (SOCKFD, sol_tcp, Tcp_ulp, "Caesar cipher", sizeof ("Caesar cipher"));
Perror ("ULP init");
Set the key for the encryption algorithm
SetSockOpt (SOCKFD, Sol_tcp, Tcp_none, &key, sizeof (int));
Perror ("cipher init");

ret = Connect (SOCKFD, (struct sockaddr *) &serveraddr, sizeof (SERVERADDR));
if (Ret < 0) {
Perror ("Connect");
Close (SOCKFD);
return 1;
}

ret = Send (SOCKFD, buf, sizeof (BUF), 0);
if (Ret < 0) {
Perror ("Recvfrom error");
Perror ("Connect");
return 1;
}

Close (SOCKFD);
return 0;
Do not look at the code of those setsockopt, just look at send/recv these, you can see and the usual and there is nothing different, the client sent to the server string "ABCDEFG", the service end is printed out is "ABCDEFG", without distortion, however, If you grab the bag, you'll see the following package:

Write a picture description here

Obviously, the data is encrypted!

?? Without a tunnel, without any additional external configuration, just one setsockopt can make the data encrypted? It's unbelievable. This is the presentation layer implemented by ULP, and we'll see how this is implemented.

Very simply, the ULP framework in the kernel actually does two things:

ULP support the framework to do things
Replace setsockopt for its own setsockopt callback, which calls the original setsockopt by default.

The concrete Ulp realizes the thing that does
Implement the setsockopt callback, replace the proto struct in the setsockopt callback, and re-implement the callback function sendmsg,sendpage,recvmsg in proto struct.

Implementation is a bit humble, but first of all, the implementation of a demo, and I will immediately give the implementation, that is, the implementation of Caesar encryption, code as follows (compiled into Caesar_cipher.ko):

#include <linux/module.h>
#include <net/tcp.h>

static struct Proto Caesar_cipher_base_prot;
static struct Proto Ulp_caesar_cipher_prot;

#define Tcp_none 100

struct Caesar_cipher_context {
Shabby keys.
int key;
Preserving the setsockopt function of the original proto structure
Int (*setsockopt) (struct sock *sk, int level, int optname, char __user *optval, unsigned int optlen);
Preserving the getsockopt function of the original proto structure
Int (*getsockopt) (struct sock *sk, int level, int optname, char __user *optval, int __user *optlen);
Preserving the sendmsg function of the original proto structure
Int (*sendmsg) (struct sock *sk, struct MSGHDR *msg, size_t len);
Preserving the Recvmsg function of the original proto structure
Int (*recvmsg) (struct sock *sk, struct MSGHDR *msg, size_t len, int noblock, int flags, int *addr_len);
Keep the close function of the original proto structure
void (*close) (struct sock *sk, long timeout);
};

Static inline struct Caesar_cipher_context *caesar_cipher_get_ctx (const struct sock *sk)
{
struct Inet_connection_sock *icsk = INET_CSK (SK);
Return icsk->icsk_ulp_data;
}

It doesn't make sense to realize this.
static int caesar_cipher_getsockopt (struct sock *sk, int level, int optname,
Char __user *optval, int __user *optlen)
{
struct Caesar_cipher_context *ctx = Caesar_cipher_get_ctx (SK);
Return ctx->getsockopt (SK, Level, optname, Optval, Optlen);
}

static int caesar_cipher_setsockopt (struct sock *sk, int level, int optname,
Char __user *optval, unsigned int optlen)
{
struct Caesar_cipher_context *ctx = Caesar_cipher_get_ctx (SK);
struct Proto *prot = NULL;
int Val;

if (level! = Tcp_none) {
Return ctx->setsockopt (SK, Level, optname, Optval, Optlen);
}

if (Get_user (Val, (int __user *) optval)) {
Ctx->key = 0;
} else {
Ctx->key = val;
}

Replace the proto of the socket and keep the original close callback
Prot = &ulp_Caesar_cipher_prot;
Ctx->close = sk->sk_prot->close;
Sk->sk_prot = prot;

return 0;
}

int caesar_cipher_sendmsg (struct sock *sk, struct MSGHDR *msg, size_t size)
{
struct Caesar_cipher_context *ctx = Caesar_cipher_get_ctx (SK);
Char *userbuf;
struct MSGHDR newmsg;
struct Iovec iov[1];
int err;

mm_segment_t Oldfs = Get_fs ();

Assign a new buffer, which is for user space and kernel space conflicts
Userbuf = vmalloc (size);
if (!USERBUF) {
Return-enomem;
}

Iov[0].iov_base = Userbuf;
Iov[0].iov_len = size;
Err = memcpy_from_msg (userbuf, MSG, size);
if (err) {
Vfree (USERBUF);
Return-einval;
}

Inline Encrypt Caesar encryption operation
{
int i = 0;
for (i = 0; i < size; i++) {
Userbuf[i] = Userbuf[i] + ctx->key;
}
}

Iov_iter_init (&newmsg.msg_iter, WRITE, Iov, 1, size);
Newmsg.msg_name = NULL;
Newmsg.msg_namelen = 0;
Newmsg.msg_control = NULL;
Newmsg.msg_controllen = 0;
Newmsg.msg_flags = msg->msg_flags;

Declaration is now sending data in the kernel state
Set_fs (Kernel_ds);
Call the original sendmsg to send the newly allocated buffer
Err = ctx->sendmsg (SK, &newmsg, size);
Set_fs (OLDFS);

Vfree (USERBUF);

return err;
}

The implementation of this callback function is no different from the sendmsg idea.
int caesar_cipher_recvmsg (struct sock *sk, struct MSGHDR *msg, size_t len, int nonblock,
int flags, int *addr_len)
{
struct Caesar_cipher_context *ctx = Caesar_cipher_get_ctx (SK);
int ret, err;

Char *userbuf;
struct MSGHDR newmsg;
struct Iovec iov[1];

mm_segment_t Oldfs = Get_fs ();

Userbuf = Vmalloc (len);
if (!USERBUF) {
Return-enomem;
}

Iov[0].iov_base = Userbuf;
Iov[0].iov_len = Len;

Iov_iter_init (&newmsg.msg_iter, READ, Iov, 1, Len);
Newmsg.msg_name = NULL;
Newmsg.msg_namelen = 0;
Newmsg.msg_control = NULL;
Newmsg.msg_controllen = 0;
Newmsg.msg_flags = msg->msg_flags;

Set_fs (Kernel_ds);
ret = ctx->recvmsg (SK, &newmsg, Len, Nonblock, Flags, Addr_len);
Set_fs (OLDFS);

Inline Decrypt Caesar decryption operation
{
int i = 0;
for (i = 0; i < ret; i++) {
Userbuf[i] = userbuf[i]-ctx->key;
}
}

Err = memcpy_to_msg (msg, USERBUF, ret);
if (err) {
ret =-einval;
}

Vfree (USERBUF);
return ret;
}

static int caesar_cipher_init (struct sock *sk)
{
struct Inet_connection_sock *icsk = INET_CSK (SK);
struct Caesar_cipher_context *ctx;
int rc = 0;

CTX = Kzalloc (sizeof (*CTX), Gfp_kernel);
if (!CTX) {
rc =-enomem;
Goto out;
}

Icsk->icsk_ulp_data = CTX;

Preserve the core operations of Set/getsockopt
Ctx->setsockopt = sk->sk_prot->setsockopt;
Ctx->getsockopt = sk->sk_prot->getsockopt;
Preserve the core operations of SEND/RECVMSG
Ctx->sendmsg = sk->sk_prot->sendmsg;
Ctx->recvmsg = sk->sk_prot->recvmsg;

The overall replacement of the socket's proto operation callback, but in fact it is copied the original proto operation Callback, the real replacement operation in setsockopt
Sk->sk_prot = &Caesar_cipher_base_prot;
Out
return RC;
}

static void Caesar_cipher_close (struct sock *sk, long timeout)
{
struct Caesar_cipher_context *ctx = Caesar_cipher_get_ctx (SK);
Kfree (CTX);
Ctx->close (SK, timeout);
}

static struct Tcp_ulp_ops Tcp_caesar_cipher_ulp_ops __read_mostly = {
. Name = "Caesar cipher",
. Owner = This_module,
. init = Caesar_cipher_init,
};

static int __init caesar_cipher_register (void)
{
Caesar_cipher_base_prot = Tcp_prot;
caesar_cipher_base_prot.setsockopt = caesar_cipher_setsockopt;
caesar_cipher_base_prot.getsockopt = caesar_cipher_getsockopt;

To replace individual callback functions
Ulp_caesar_cipher_prot = Caesar_cipher_base_prot;
Ulp_caesar_cipher_prot.sendmsg = caesar_cipher_sendmsg;
Ulp_caesar_cipher_prot.recvmsg = caesar_cipher_recvmsg;
Ulp_caesar_cipher_prot.close = Caesar_cipher_close;

This registration is no longer explained, nothing more than to link a struct to a list, wait until setsockopt performs ULP binding and then finds it.
Tcp_register_ulp (&tcp_caesar_cipher_ulp_ops);

return 0;
}

static void __exit caesar_cipher_unregister (void)
{
Tcp_unregister_ulp (&tcp_caesar_cipher_ulp_ops);
}

Module_init (Caesar_cipher_register);
Module_exit (Caesar_cipher_unregister);

Module_author ("Zhao Ya <[email protected]>");
Module_description ("Linux 4.14 ULP Demo");
Module_license ("GPL");
Based on the above description and source code, it is easy to understand why two setsockopt are required to complete the registration and binding. The first setsockopt just replaced the setsockopt function, the second setsockopt replaced the partial proto operation callback and set the encryption key, and then the new proto operation was free to sway, and that was it.

?? If you look closely at the realization of KTLS, it is only that. The beauty is ULP and there is no other. However, I am not saying that it is possible to seamlessly encrypt the application data? If the application can not be changed, but also say that can not be added to the two setsockopt, then what? Simple! Use the Linux ld_preload to hijack the system call, just take the server as an example. Please comment out the setsockopt in the C code of the server above, and compile the following code:

#define _gnu_source
#include <stdio.h>
#include <dlfcn.h>
#include <errno.h>
#include <netinet/tcp.h>

#define TCP_ULP 31
#define Tcp_none 100

int key = 12354;

int accept (int sockfd, struct sockaddr *addr, socklen_t *addrlen)
{
int CLISD;
typeof (Accept) *_accept;

_accept = Dlsym (Rtld_next, "accept");
CLISD = (*_accept) (SOCKFD, addr, Addrlen);
SetSockOpt (CLISD, sol_tcp, Tcp_ulp, "Caesar cipher", sizeof ("Caesar cipher"));
Perror ("Client ULP init");
SetSockOpt (CLISD, Sol_tcp, Tcp_none, &key, sizeof (int));
Perror ("Client cipher init");

return CLISD;
The Compile method is as follows:

Gcc-shared-fpic Preload_ulp.c-o preload_ulp.so
1
Then, before running the server, refer to an environment variable:

Ld_preload=./preload_ulp.so./server
1
Can!

?? Well, it's now possible to run them uniformly. First load the ULP kernel module:

Insmod./caesar_cipher.ko
1
Then execute the server:

Ld_preload=./preload_ulp.so./server
1
Last execution client:

./client
1
OK, the service side perfectly prints out "ABCDEFG". Is it over? It's over!

Now to judge it, Ulp really a little low, based on its low, I can only achieve a Caesar encryption, the more large implementation I stopped at those complex data structure analysis, of course, this may be because I do not fine, but this does not also confirm the Ulp interface unfriendly? A friendly interface allows people to program like a boiler, like a meal, when an interface call is like a pull-up, it's sad!
...

?? My feeling is, and my desire is, to achieve a iptables target that enables seamless Caesar encryption on an end-to-end for any existing connection, such as:

Iptables-a input-s 1.1.1.1-p tcp--sport 123-d 2.2.2.2--dport 234-j gaiusdecrypt--key 12354
Iptables-a output-s 1.1.1.1-p tcp--sport 123-d 2.2.2.2--dport 234-j gaiusdecrypt--key 12354
1
2
Then the corresponding five-tuple tcp-1.1.1.1:123/2.2.2.2:234 traffic is encrypted by the key 12354, which is more handsome, all you need to do is to find the socket in the target processing function and then call setsockopt, Or more simply, find the conntrack and set the key to the Conntrack table entry. However, this is a very simple idea, more complex gameplay in the imagination, completely unrestricted.

Process

Writing this Caesar cipher kernel module is hard, this is not to say ULP principle is how difficult, code how difficult to write, in fact the code is very simple, the problem is in the environment of the building.

?? I have a Debian 9 development machine, it has built up 4.9 of the kernel, is enough new, but I'm sorry, ULP and KTLS are only supported on 4.13+, this from the following link can be seen:
Linux 4.13#networking:https://kernelnewbies.org/linux_4.13
So I chose the 4.14 version for the demo development. I have allocated 4G of disk space for the source code compilation, using the Debian 9 own config file to compile, but not long before the hint of space, I can only re, this time I increased the space by one times, to reach the appearance of 8G, so I went to sleep, the next day woke up, A few hint of the lack of space in the big print blind my eyes ... As I saw already to the last step, so I increased the space to 9G, after opening the script to go to work, at night to get home, finally took care of .... This just started to write this demo, the lack of space problem full of my day and night time!

?? I wonder when it takes so much disk space to compile a kernel, is the distribution really getting bloated? How do you play it? Why not lose weight when you are so fat?

ULP (Upper Layer Protocol) brought in by the Linux 4.13/4.14 kernel