Took some time to study the next ktls, which stems from a problem with colleagues, that is, the HTTPS server today and the SCP command to transfer local files, can not use the Sendfile system call!
This topic reminds me of a lot of old colleagues, in order not to harass them, in this article I will use only we know the nickname.
I regret that I did not think of a way to let HTTPS/SCP support sendfile/splice/tee call family, I regret because this is more than OpenVPN to reflect the industry's influence and personal value, whether for the company or the individual, will be of infinite value. And, this is a fusion of system calls, network technology, encryption and decryption technology in one direction.
I experienced a second confusion in the beginning of 2010 after a year, struggling to find the direction, in my head into the half bottle of OpenVPN details and smattering the original version of the standard ECC encryption algorithm and the fusion of OpenSSL, I chose OpenVPN, Then I said no more in the SSL protocol breakthrough, a go is four or five years! I could have done something like AF_KTLS, but I've brushed it off again and again, until now, I've left the company, and I'm not even doing anything related to SSL/TLS or PKI, but suddenly let me know there's a ready-made solution from Facebook, It's been 2016 years! Facebook's program came from 2015, but we can actually do it earlier, but no, so I'm very sorry! The reason I'm saying this is because I almost don't need you to think about it. Step by step keep up with Dave Watson's ideas, everything seems so natural and direct, like KTLS is a matter of course, I have published the "OpenVPN Linux kernel version, ghost of the incomplete" This series of articles, and did the same thing, but a little bit, if I was not in the optimization of the VPN, but in the optimization of the forward proxy or reverse proxy, or in the optimization of the HTTPS server itself, then I think the kernel should be moved into the TLS record protocol. I extract a quote from the article "OpenVPN Linux kernel version, ghost of the broken part iv:normal Method" words:
1. Kernel module: Run a kernel-mode TCP server, after receiving data according to the value of the first byte of the packet is to control the channel data or data channel data, if the control channel data, through a character device routed to the user state of a process, if it is data channel data, is processed directly in the kernel;
It was in 2014 years!
Another episode that was, in 2010, I didn't know that OpenVPN had its own encapsulation protocol, and I always thought that OpenVPN was using an SSL recording protocol to encapsulate the data, which was obviously the first false perception of OpenVPN, But if I've been wrong and I've been wrong until 2014 when porting the data channel processing logic to the kernel, I still throw the TLS logging protocol into the kernel. Although, I used the NetFilter module mechanism rather than the socket mechanism ...
Before the text, fan point, Wenzhou leather shoes factory boss, Wang sister, light rain, small group, wood manager, lead guitarist, Uncle Wah ... Alas, nine taste miscellaneous chen Ah ...
In addition, KTLS has been out for more than a year, the online resources are very limited, in addition to Google can find a few patches is from Dave Watson's GitHub itself, the Chinese community almost did not discuss this, Baidu search results are among insects, so I wrote this article, Just like the popular Science articles on TCP BBR and TCP CDG, I feel that I have once again occupied the sofa.
The above is a routine feeling, no regrets is unwritten, each article is an emotional burst, coupled with the middle of the night to find the day there is rain, no longer sleep, I am more like rain.
-----------------------------------
We know that Sendfile can allow the file system to communicate directly with another file system, in terms of the communication network, the user program is only in the control plane, the data plane is completely in the kernel, so as to avoid the memory copy and a variety of switching, which greatly improve performance. The general Web server or file server is the use of Sendfile to send a file directly to a socket, but for HTTPS, this is not possible, because the kernel can not handle encryption, you always have to pull the data to the user state, encrypt it, and then encapsulated into the TLS record protocol, Then write to the socket ...
Wait! Don't you have a af_alg socket? Yes, this is one of the contents of this article, it is not time to explain.
Not only HTTPS, but also the system development or operation and maintenance personnel on weekdays are very commonly used in SSH suite SCP, can not use sendfile, the reason ditto! We can see from the figure below.
The same idea, see a description of the BSD system optimized HTTPS "optimizing TLS for High–bandwidth applications in FreeBSD". However, this article is not intended to detail sendfile/splice/tee This system call family, these are just a primer, I would like to launch a discussion of the kernel-State TLS. The structure of this article is broadly as follows:
0. A routine feeling
1. Introduction to the principle of KTLS
2. Show me how to run Dave's example
3. Simple evaluation of the KTLS
4. Another alternative option
-----------------------------------
This time Facebook brings the gospel, see "TLS in the kernel", Dave Watson from Facebook introduced a new socket, AF_KTLS, which can be attached to a tcp/ The UDP sockets are directly encrypted in the kernel state and do the TLS encapsulation, as shown in the framework:
As described by the author, the TLS handshake logic is still done in the user state, and this handshake protocol is actually a matter of control plane. In the AF_KTLS socket, in addition to completing the encryption/decryption and Recording protocol encapsulation/unpacking, all other operations are done by the TCP/UDP socket attached to it, so the AF_KTLS socket exposed to the user interface is the standard af_inet Socket TCP/UDP socket, you can of course send the file by Sendfile!
The principle is introduced here, very simple! Complex things lie in the encryption, decryption, HMAC those operations, this aspect I am not an expert, in the PKI company for more than 5 years more than just understand, so that is not here to install forced. Wenzhou boss claims more than I know, but also just called, Wang sister that is really understand.
-----------------------------------
Let's take a look at how to get Dave's example running.
1.KTLS Source Download
We downlaod the source code down from Ktls's GitHub and compile it with the following address: Https://github.com/ktls/af_ktls
2. Compiling and loading
This step I divided into the following several steps, for the familiar person, super simple.
2.1. Select the kernel version
I chose the latest Linux 4.9 kernel, just now the BBR algorithm, is also used in this kernel. I'm going to say good-bye to the 2.6.32 kernel after this.
2.2. Patching and restarting to the new kernel
Go to the 4.9 kernel root directory and execute directly
Patch-p1 < $RFC 5288.patchPath, and then routinelyMake
all-j20 && make modules_install && make installCan. After the compilation succeeds, reboot to the new kernel and schedule the new 4.9 kernel header file path to prepare the module for compilation.
2.3. Compiling kernel modules
At this point start compiling af_ktls.c, direct make can, skb_splice_bits this function parameters will be reported too many, according to 4.9 of the kernel fix. Then load the compiled generated Af_ktls.ko.
3.KTLS instance Download
Dave, by the way, wrote a very simple example of using KTLS to record protocol encapsulation, based on OpenSSL. The structure of this example is very simple:
Server Thread: Run a OpenSSL-based server, call Ssl_write/ssl_read in a loop to read and write data, and output the results;
Client thread: Runs a OpenSSL-based clients, first using the standard ssl_read/ssl_write for data read and write, then calls the AF_ALG socket for data read and write, and observes the results of the output on the server Thread.
Note: Dave's demo is using the AF_ALG socket instead of the AF_KTLS, and at first I was wrong with him, and later found out that it was two versions.
It has a TLS server and a TLS client, and after the TLS client and TLS server have successfully completed the TLS handshake, the client will call Ssl_read/ssl_write with standard OpenSSL and KTLS socket send/recv/ Sendfile two ways to communicate with the server, to prove that the effect of the two methods is equivalent, so as to illustrate the KTLS implementation of the Protocol compatibility with OpenSSL.
The GitHub address is as follows: Https://github.com/djwatson/ktls. As usual, we should download it locally and compile the run.
-----------------------------------
Unfortunately, however, this example is not implemented for the AF_KTLS socket above, but for another equivalent mechanism based on the AF_ALG socket (Algif_tls patch, later talks). I don't know why Dave was re-af_ktls the demo on the basis of the AF_KTLS, although the OpenSSL-based example was not done, but Dave completed a more formal example based on GUNTLS, with the following address: https:// Github.com/ktls/af_ktls-tool. By compiling and running this example, you will find that Ktls is indeed correct and efficient. However, I believe that it is more familiar with OpenSSL than GnuTLS. And, Ktls This example is very simple, just a C file, this style is my favorite, simple and clear, do not have to fall into the code structure, language and tool details, so I would like to complete the above based on the OpenSSL test af_ktls example.
Then I took some time and finally changed the example of Dave's Test Algif_tls to the example of the test af_ktls and switched it, and fork it out, the address in: Https://github.com/marywangran/ktls/tree/patch-1.
The process of debugging is very tedious, mainly "how to remove key,iv and salt problems from the EVP_AES_GCM_CTX structure of OpenSSL", which is done under the advice and guidance of sister Wang. I have to say, debugging encryption algorithm is very complex things, this kind of thing I have done, but not long, and sister Wang is the veteran of this field! Even for Dave Watson, it's probably because of the encryption algorithm that the HAMC algorithm is so complicated that it has to start out with a simplified version that only supports GCM (AES) 128bit ... But listen to Sister Wang said, RFC5288 described the AES Galois Counter mode (GCM) Cipher Suites for TLS is much faster than CBC mode, and security is equal, and no longer need to independently compute the HMAC, I believe Dave also understand this. So the so-called "simplified version" of Dave's Ktls is not because it's easy to implement.
This is the end of the encryption algorithm, take some time this weekend to study the principle of GCM.
----------------
My modification is mainly to change the operation of AF_ALG socket into the AF_KTLS socket operation:
....//equivalent OpenSSL operation/* Kernel TLS Tests */INT TFMFD = socket (af_ktls, sock_stream, 0); if (TFMFD = =-1) {perror ("Socket Error:"); Exit (-1); } struct Sockaddr_ktls sa = {. Sa_cipher = ktls_cipher_aes_gcm_128,/* Specifies cipher suit*/. SA _socket = Server,/* Specifies the attached TCP socket */. Sa_version = Ktls_version_1_2,}; if (Bind (TFMFD, (struct sockaddr *) &sa, sizeof (sa)) = =-1) {perror ("Af_alg:bind failed"); Close (TFMFD); Exit (-1); } evp_cipher_ctx * Writectx = ssl->enc_write_ctx; Evp_cipher_ctx * Readctx = ssl->enc_read_ctx; evp_aes_gcm_ctx* gcmwrite = (evp_aes_gcm_ctx*) (writectx->cipher_data); evp_aes_gcm_ctx* Gcmread = (evp_aes_gcm_ctx*) (readctx->cipher_data); unsigned char* Writekey = (unsigned char*) (Gcmwrite->gcm.key); unsigned char* ReadKey = (unsigned char*) (Gcmread->gcm.key); unsigned char* writeiv = gcmwrite->iv; unsigned char* readiv = gcmread->iv; Char keyiv[20] = {0}; memcpy (Keyiv, Writekey, 16); if (setsockopt (TFMFD, Af_ktls, Ktls_set_key_send, Keyiv, +)) {perror ("Af_alg:set write KEY failed\n"); Exit (-1); } memcpy (Keyiv, Writeiv, 4); if (setsockopt (TFMFD, Af_ktls, Ktls_set_salt_send, Keyiv, 4)) {perror ("Af_alg:set write IV failed\n"); Exit (-1); } uint64_t Writeseq; unsigned char* writeseqnum = ssl->s3->write_sequence; memcpy (&writeseq, Writeseqnum, sizeof (WRITESEQ)); if (setsockopt (TFMFD, Af_ktls, Ktls_set_iv_send, (unsigned char*) &writeseq, 8)) {perror ("Af_alg:set write salt FA Iled\n "); Exit (-1); } memcpy (Keyiv, ReadKey, 16); if (setsockopt (TFMFD, Af_ktls, Ktls_set_key_recv, Keyiv, +)) {perror ("Af_alg:set read KEY failed\n"); Exit (-1); } memcpy (Keyiv, Readiv, 4); if (setsockopt (TFMFD, Af_ktls, Ktls_set_salt_recv, Keyiv, 4)) {perror ("Af_alg:set Read IV failed\n"); Exit (-1); } uint64_t Readseq; unsigned char* readseqnum = ssl->s3->read_sequence; memcpy (&readseq, Readseqnum, sizEOF (READSEQ)); if (setsockopt (TFMFD, Af_ktls, Ktls_set_iv_recv, (unsigned char*) &readseq, 8)) {perror ("af_alg:set read Salt fail Ed\n "); Exit (-1); } ....//Send/recv/sendfile on the TFMFD socket, the TFMFD socket completes record encryption decryption and encapsulates the encapsulation of the Protocol, interacting with the tcp/udp socket beneath it.
-----------------------------------
Before continuing with another AF_ALG-based ktls scenario, I'll put a few comments here.
I personally feel that KTLS is not just a patch to optimize TLS, but rather a veritable TLS, the Transport Layer Security! Prior to KTLS, packages based on user-state libraries such as OpenSSL could only be "buffer safe", not true transport Layer security, since the encapsulated data was just a socket buffer. I personally pay more attention to the overall architecture, so I am here to do the semantics.
Maybe it's because Facebook doesn't have the clout of Google, and Dave Watson is a low-key person, so ktls didn't attract a puff of hype or even blindly follow suit, but the sound of questioning is constantly, in a sense, a good thing. Is it really necessary to think calmly about doing this, does it really make sense to put the TLS protocol in the kernel?
True, Dave Watson does not recommend that the entire TLS protocol be implemented at the kernel level, but only for symmetric cryptographic operations and for a fixed and stable operation of encapsulation operations. The reason that symmetric encryption and encapsulation operations are unloaded is because such operations are "fixed set of serialized operations". This is the same as TCP offloading the checksum operation and staging operations to the network card without offloading the TCP three handshake and congestion control to the network card, and people have to weigh the complexity and efficiency of the implementation and find a balance between the two before they can achieve the goal of optimization by means of down-unloading. Another consideration is how much the upgrade will cost if the part of the uninstallation has changed on the standard.
The TLS Protocol and TCP protocol development very similar, if willing, you can also divide the TCP protocol into the Handshake Protocol and the Record Protocol, and the TCP computation check code as well as the fragment, the TLS recording protocol part of the operation is very fixed, aes/des this symmetric encryption algorithm is very stable and fixed, It's not going to change for a long time, and the difference is that the TLS handshake and the processing of certificates are both complex and changeable. As the author hopes, Ktls hopes to take the symmetric operation from the user-State Library of the Bang of the wind, and finally push the network card manufacturers to implement this hardware offload operation, the symmetric encryption and TLS recording protocol encapsulation operation is completely uninstalled to the network card!
In any case, ktls may not be enough, at the end of the article "TLS in the kernel", there was this passage:
Overall, it looks like it'll take some more convincing arguments before putting TLS in the kernel would be seriously Considered. For some specialized situations, it might make sense to does, but even the limited version Watson posted adds more than 1 Lines of code to the kernel-for dubious gains. Over time, more and more crypto have been added to the kernel, though, so maybe TLS would eventually find its-in too.
I and Wenzhou leather shoes factory boss of the dialogue is not affixed, Wenzhou boss especially want to engage in a such project to make a point of influence, I said I will fully support.
----------------
I personally agree with Dave Watson's point of view. But have you ever thought that assuming this ktls is applied, what will have to change?! Need to change is Apache,nginx this mainstream server's HTTPS processing logic, will all call Ssl_read/write place to Ktls-based recv/send/sendfile, undoubtedly, the price of change is very huge!
My praise for Ktls's simplicity may come from my prejudice against OpenSSL, which falls within the realm of personal emotion and does not incite.
As security becomes more and more important, encrypted communication is no longer an additional component, it will become part of the network protocol, which can be seen from the gradual popularity of HTTPS, "buffer security" or "Transport Layer Security", which is your pursuit? Please keep glimpse!
-----------------------------------
Now it's time to talk about another option.
To achieve KTLS, in addition to the direct implementation of a AF_KTLS socket like the above, there is a solution, is directly based on AF_ALG socket to achieve, so that the AF_ALG socket can do sendfile fd_in parameters! So
In fact, this is also the ALGIF_TLS scenario for Dave Watson, and the patch for the program and the patchwork address for the discussion are: https://patchwork.kernel.org/patch/7684551/
To understand the rationality of this approach, I need to take a bit of space to introduce the AF_ALG mechanism of a Linux kernel.
Af_alg is a type of socket that is tied to Af_inet, which is designed to export a family of user interfaces that an application can encrypt and decrypt by invoking the socket interface.
For an initial patch introduction to AF_ALG, take a look at the LWN article Crypto:af_alg-user-space interface for Crypto API. If you have studied the code, it is not difficult to figure out the relationship shown:
First of all, we can see that the first AF_ALG socket built through the socket is just a socket set up crypt mechanism, and finally need to call accept to get an operation socket, this process is very similar to the TCP and Linux init process! Second, we notice that the af_alg_type is very important, and its existence is intended to allow you to implement different crypto APIs, such as invoking different cryptographic algorithms, but it also makes it possible to customize any operation, and you can make a af_alg The socket only completes the encryption and decryption when calling send, then exports the result when recv, or attaches a af_alg OP socket to a TCP socket or UDP socket, and when the send is called, the encryption is done first. The result is then sent to the network via its attached TCP socket or UDP socket, depending on how you implement the callback in the Af_alg_type.
The AF_ALG frame socket itself is an abstract framework, and the specific work is done by linking it to a af_alg_type, which is generated by an OP socket. This principle provides the basis for Dave to implement the patch of ALGIF_TLS, in fact, he is implementing a ALGIF_TLS structure:
static struct Proto_ops algif_tls_ops = {. Family = Pf_alg, ...//sendmsg/page 1. Encryption 2. Encapsulating messages through the TLS logging protocol; 3. Send to the underlying attached tcp/udp socket. sendmsg = tls_sendmsg,. sendpage = Tls_ Sendpage,//recvmsg 1. Obtain data for the underlying attached TCP/UDP socket, 2. Unpack the data, 3. Decrypt the data and copy it to the user state buffer. recvmsg = tls_recvmsg ,. Poll = sock_no_poll,};static const struct Af_alg_type algif_type_tls = {//bing create the underlying CRYP To_aead infrastructure. Bind = Tls_bind,. Release = Tls_release,//Setkey at the bottom crypt O_aead set the key on the facility. Setkey = Tls_setkey,. setauthsize = tls_setauthsize,//Accept Gets an op socket for the actual read and write operation, and the following OPS field as its action callback collection. Accept = Tls_accept_parent,//OPS actual OP socket callback Collection. Ops = &algif_tls_ops,//name can be found by this name when the frame socket is bind, and the algif_tls_ops is used for the cable Cited. Name = "TLS",. Owner = this_module};
Let's take a look at Dave's tls.c as an example of how to do this:
/* Kernel TLS Tests */// first create frame socket int TFMFD = socket (af_alg, sock_seqpacket, 0); Create an index structure to find the Af_alg_type struct sockaddr_alg sa = { . salg_family = Af_alg, . Salg_type = "TLS" registered in the kernel,/ * This selects the hash logic in the kernel * /. Salg_name = "rfc5288 (GCM (AES))"/* This is the cipher name */ };
//based on the above SA to establish a frame socket associated with Af_alg_type if (Bind (TFMFD, (struct sockaddr *) &sa, sizeof (sa)) = = 1) // Get an op socket int opfd = Accept (TFMFD, NULL, 0) for the actual IO operation; ...//The following OP socket behavior is entirely dependent on the OPS field indicated in the Af_alg_type proto_ops back to the mobilized.
The above is the framework of the AF_ALG socket and another KTLS scenario that Dave Watson implemented on this framework differs from AF_KTLS. But this is not all I want to say, since AF_ALG's framework is so flexible, why do you have to do a new socket type AF_KTLS? This may be a metaphysical question, and I will extract the content from this article as follows, from Af_alg author Herbert Xu:
On Mon, Nov at 09:43:02am-0800, Dave Watson wrote:
> Userspace crypto interface for TLS. Currently supports GCM (AES) 128bit only,
> However the interface is the same as the rest of the Sock_alg interface, so it
> should is possible to add more without any user interface changes.
SOCK_ALG exists to export crypto algorithms to user-space. So if
We decided to support TLS as a algorithm then I guess this makes
Sense.
However, I must say that it wouldn ' t has been my first pick. I ' d
Imagine a TLS socket to look more like a TCP socket, or perhaps a
KCM socket as proposed by Tom.
So, Af_ktls is a more reasonable choice?
Now continue to Af_ktls and af_alg+ to customize the af_alg_type of the merits of the metaphysical discussion. Personally think or af_alg+ custom af_alg_type more flexible, you just need to implement a af_alg_type and register into the kernel, you can implement any form of encapsulation, and even implement a complete HTTP protocol processing. It's not entirely like AF_KTLS to implement a full socket type yourself, which would put you in a similar quagmire like the IOCTL distribution. Admittedly, Sendfile kicked the kernel out of the 2.4 kernel's kernel Web server, but af_alg+ custom Af_alg_type is a more generic mechanism that can implement any application-level protocol!
-----------------------------------
Remember that I ported the OpenVPN Data channel protocol to the kernel state from 2014 to 2015, GitHub address: Https://github.com/marywangran/OpenVPN-Linux-kernel
This work is very similar to Dave Watson's work, after careful study of Dave's Ktls, I wanted to spend the weekend to re-structure the Kopenvpn again, but obviously feel powerless, so I retreated to the second, I just want to customize af_ based on af_alg+ Alg_type to implement a simple HTTP protocol, only to transfer files, the general logic is very simple, that is, through sendfile send a file to a af_alg OP socket, and then implement a Af_alg_ The type finishes adding HTTP headers before continuing to the underlying inet socket. In a different way, I can also write a separate af_myhttp module to use a new socket more directly to achieve the above requirements.
------------------------------------
After finishing this article, my next challenge is to start singing "Rock and Roll on the new Long March Road" in the small kindergarten's community open space on December 31, 2016 (which came from a little autumn outing a few weeks ago on the way I was forced by a madman to sing a "fake Monk", Then kindergarten teachers have to let me in the new Year's party to another rock ...) ... A little nervous, decided to drink a bottle of wine again! Fortunately the madman can accompany me to sing "One Night in Beijing" .... Someone to accompany me ashamed, is always more embarrassing than a person ... Anyway, come on! For children, anything can pay, sing a song to lose a person count a wool ah!
KTLS (Kernel SSL/TLS) principles and examples from Facebook
