This is a creation in Article, where the information may have evolved or changed.
"Editor's note" This share focuses on the design and implementation of the open-source PaaS platform Lain, based on the WebSocket and Docker remote APIs, to the single-process container function remotely in the trusted Big Data Innovation Center.
"Shanghai station |3 Day burn Brain Service Architecture training Camp" Training content includes: DevOps, MicroServices, Spring Cloud, Eureka, Ribbon, feign, Hystrix, Zuul, Spring Cloud Config, spring Cloud sleuth and so on.
Lain Platform Introduction
Lain is a PaaS platform based on Docker developed by the Big Data Innovation Center. It is now open source to GitHub and has been internally produced.
Lain provides a holistic solution for DevOps issues on bare metal and under application development, standardizing application development, testing, on-line workflow, supplemented by SCM support, improved overall resource utilization, and optimized redundant resource pools.
Lain implements Docker clustering through Swarm, using ETCD as the service discovery and the underlying metadata store. We also integrate our own development of container scheduler, controller and Rights management system.
Research on remote access container scheme
It is believed that the students who have developed the PaaS platform have faced the requirement of remote access to the container debugging by the platform users.
In a multi-process container scenario, this requirement seems to be easy to implement, just to start an sshd process internally. However, this design poses some problems.
First, permissions management. How do I control the permissions correspondence between users and containers? In a traditional virtual machine or physical server usage scenario, the company will be equipped with the appropriate bastion management system for these infrastructures. But equipping a PaaS platform with such a system can complicate and bloat the platform.
The second is configuration management. If the user needs to change the password or trust the public key, how can the sshd configuration of all the containers be synchronously modified and guaranteed to be consistent?
The last is container process management. The container's No. 1th process requires not only the management of sshd, but also the need for the sshd to expose the port to the client connection. Due to the differences in the environment of different host hosts, it is likely that there will be inconsistencies in the exposed ports, which can also severely affect the user experience.
In a single-process container scenario, the above scenario is not feasible. Allowing users to log on to the host directly to perform Docker exec-it is obviously not a viable option either.
By investigating the Docker API and its implementation, we explored a solution for container remote login requirements.
Entry applications
Entry applications are divided into server and client parts, and the overall structure is as follows.
The Entry server is responsible for communicating with the Docker process, obtaining the output of the terminal in the container and sending the user input to the terminal. The Entry server is also responsible for user authentication, receiving user input from the terminal and returning the output of the container terminal to the user.
Entry's client is divided into command-line clients and WEB clients, both of which function similarly, are responsible for connecting the service side, and send the user's keyboard input, receive and simulate the terminal output of the container.
Entry service-side and Docker communication implementations
Before we introduce the implementation of server and Docker communication, let's look at one of Docker's APIs.
As you can see, the API implements full-duplex transmission of inputs and outputs through WebSocket.
Inspired by this API, we believe that since the input and output of the container can be transmitted via WebSocket, it is also possible to transmit the user's input and output naturally. Finally, the two parts of the input and output docking can be achieved.
The advantage of using WebSocket is that because the WebSocket is connected via the HTTP protocol, the data transfer can take advantage of the unified Nginx portal without the need for additional open ports. In the development process, we can define our own message format on the basis of the WebSocket protocol, which makes our application design more flexible.
There are already many good Docker client libraries and WebSocket libraries in the Go language. We are using github.com/fsouza/go-dockerclient and Github.com/gorilla/websocket.
There are two APIs associated with Docker exec, the first of which is the createexec.
The corresponding parameters of the structure of the body such as:
In the parameters, both Attachstdout,attachstderr,attachstdin and Tty are set to true, that is, we need to open the terminal and get its input and output stream. The cmd parameter is set to the bash process behind the "Docker exec-it XXX bash" that we often use. Because Docker's own terminal type (dumb) is too bad, the staging environment variable term=xterm-256 is set to improve the experience when you execute the bash command.
The second API is startexec.
The structure of its corresponding parameters such as.
After successful execution of Createexec, the corresponding execid will be generated. The parameters also need to be passed to Stdout/stderr and StdIn, respectively, to write to or read from writer and reader. And we need to read or write data from the other side, which is like both ends of the pipeline.
Exactly, the Go language provides Pipereader and pipewriter similar to pipelines. Via IO. Pipe () to get a pair of writer and reader. Then, start 3 goroutine to handle the input and output of these three types of data, so that the service-side and Docker communication is basically done.
Entry Server and client communication implementations
The Entry server is essentially a WebSocket server that accepts WebSocket requests from clients and transmits data through WebSocket. In high-level languages, operations on WebSocket are in the message unit. We can define the protocol format for the data within each message.
Entry, use PROTOBUF3 to define the serialization protocol for the data. Such as.
Requestmessage sends request information on behalf of the client, including two types of PLAIN and WINCH.
PLAIN represents the user's general input information. The WINCH represents the size change request for the terminal window.
Because of the different size of the terminal window, the results are not the same, so we need to change the display of the terminal according to the user's actual situation. Fortunately, Docker provides Resizeexectty, a window-size-changing API that meets our needs.
In the PLAIN message, the content is an array of bytes entered by the user. In WINCH, the content is a byte array containing a JSON string of terminal length and width.
After receiving the PLAIN message, the server will send it to Docker intact. After the WINCH message is received, the JSON is parsed and the interface to change the window size is called. It is important to note that after Docker 1.12, only startexec is executed successfully before Resizeexectty can be executed.
Responsemessage represents the information returned by the server, including the Stdout,stderr, CLOSE and PING four types.
- STDOUT: carries the standard output data of the container terminal.
- STDERR: The standard error data for the container terminal is carried.
- Close: Is the message that the service-side connection is closed.
- PING: Is the information that the server sends to keep the WebSocket connection alive.
Terminal display effects, such as color, progress bar, etc., are achieved through a series of terminal control characters. These are also ASCII characters in nature. So when the client outputs, there is no need to think about the data, just print directly.
The following describes the workflow of client and server-side communication.
First, the client sends a WebSocket connection request to the server. For the command line client, the token information for the client (for authentication) and the container information to log in are carried in the request header. For WEB clients, because the WebSocket client in the browser does not support a custom header, it is necessary to send the authentication information in a different way. I'll talk about that later.
After the server receives the connection establishment request, determines if the command line client, the token and container information is taken from the beginning, in the permission system to determine whether the user has permission to log into the container. If there is no permission, the connection is closed and the close information is returned, and the reason for the error is brought in the message. In the case of a Web client, the server waits for the first PLAIN message sent by the client, which contains the container information and token to be logged in. The server will then do the same authentication work.
When authentication is complete, the server maintains a connection with the client and is responsible for transmitting the input and output of the client and Docker. A PING message is sent to the client at a certain time to ensure that the connection is alive.
When the client disconnects, the server disconnects from the Docker and reclaims the appropriate IO resources.
When Docker disconnects first, the server sends the close information to the client and closes the WebSocket connection while reclaiming the appropriate IO resources.
In the process of data transmission, considering that the transmitted data may contain non-ASCII characters such as Chinese, our uniform character encoding is UTF-8. However, the send buffer size used when transmitting to the client is limited.
To ensure that each sent data conforms to the UTF-8 encoding rules, Entry uses the following algorithm to calculate the longest legal UTF-8 encoded prefix position in a byte array.
Since a valid UTF-8 code has a maximum of 4 bytes, it is possible to loop 4 times to find the beginning byte of the last UTF-8.
After obtaining the longest legal UTF-8 encoded subarray, the data of the Subarray is sent to the client in the message, and the remaining bytes are placed in the buffer to start the next send.
As we can see from this process, Entry is actually a high-level agent for both terminal and Docker.
Entry's command-line client implementation
Entry command line client integrated into Lain command-line tool LAIN-CLI, the Lain enter command specifies the cluster name, app name, and container number to Telnet, using the same experience as SSH login.
Entry's command-line client uses Python implementations, and the PROTOBUF3 definition can automatically generate the corresponding Python code with PROTOBUF3.
In the implementation of the command line, after establishing the WebSocket connection, the client will first get the size of the current terminal window and send a WINCH information so that it can fit the current terminal size at the beginning. In the process of interaction, in addition to handling keyboard input, WebSocket return, the client also needs to listen to the window size Change event, which is reflected in the program as signal. Sigwinch system signal. When the signal is received, the client needs to get the updated window size again and send the WINCH message to the server.
Entry's WEB Client implementation
It is very difficult to fully emulate the command-line terminal in a Web page, and if you develop the plugin yourself, you need to handle various terminal control characters.
Here we recommend a very useful analog terminal JS Library, Xterm.js. The installation and use of this library does not require any third-party dependencies, can be introduced in the form of JS files in the page, or can be installed through NPM.
The implementation of the WEB client is similar to the command line and is also to listen for keyboard input, WebSocket return, and window changes. When listening for keyboard input and window changes, Xterm.js provides data and resize two events, so you only need to implement the handler of both events.
In the browser, pressing the ESC key loses the focus of the current component. However, at the command line, we will certainly encounter situations where the ESC key is required, such as the switch from INSERT to NORMAL in vim. Therefore, the ESC key event needs to be handled specially when implemented.
It is important to note when serializing and deserializing data because the Content field previously defined in PROTOBUF3 is a byte array type. The Web client is unable to generate code using PROTOBUF3. So here we need to use JSON instead of PROTOBUF3 serialization messages. In the Go language, JSON serialization of byte arrays translates them into Base64 encoded strings. Therefore, when sending data, the Web client also needs to convert the input into a BASE64 encoded string so that the message can be deserialized correctly on the service side. Similarly, when you receive a return message for WebSocket, you also need to decode the Base64 to get the correct output.
Because the server needs to handle requests from two different clients, we abstract the logic of serialization and deserialization, defining our own serialization and deserialization function types. In the Go language, JSON is inherently satisfied with that type.
Then packaged the implementation of the PROTOBUF.
These are the implementation details of the WEB client.
Follow-up work of Entry
There are still some problems with the current Entry.
The first is browser compatibility issues. Currently testing Web client in Chrome and Firefox browser is working properly. But in the Safari browser, there will be a stuck situation, this reason is still in the positioning.
The second is the problem of non-normal exit. If the client does not pass the exit command, but instead directly disconnects the WebSocket connection, there will be a bash process remaining in the container. A bash process can be a bit of a resource. If Docker can provide stopexec similar interfaces in the future, this problem should be solved.
Currently, we are developing Entry's new feature attach, attach to the standard output of the container, providing a remote access to the standard output of the process. The dependent interface is attachtocontainernonblocking, and the other implementations can reuse the part of enter.
Summarize
As a high-level communication agent for remote terminals and Docker, Entry provides authentication, input and output data transfer and other functions, and is a solution for remote login containers. The scheme is suitable for both single-container and multi-container scenarios.
However, the remote login container should also be used only for on-line debugging, not the same as using virtual machines. A more perfect log collection system, monitoring alarm mechanism is the key to the PaaS platform debugging and monitoring the operation of online services.
This is what I share, and you are welcome to discuss the issue together. Thank you.
Questions and Answers
Q: How is the process of server iteration upgrade guaranteed to be compatible with older versions of the client?
A: When a server upgrade is made, the agreement is not modified and only the protocol is added. Because protocol definitions are consistent, modifications to the business logic on top of the protocol do not affect the client's use.
Q: Is it possible to interfere with each other if multiple users are allowed to access at the same time?
A: Different user and server connections must be built with different WebSocket connections, and a bash process that executes two exec from the same container. But gorilla/websocket this library, for the same websocket connection, can not have more than two goroutine at the same time to read or write simultaneously, so the implementation should pay attention to concurrency control.
Q: How do the containers for different hosts communicate?
A:entry is used as a platform user (application development Ops) to telnet from its own machine. This question should belong to the platform network design part, our platform uses the Calico Realization Container network, here does not unfold the discussion.
Q: Can tokens be tampered with during the connection?
A: Even if the client forges tokens, the token cannot be logged in to the container if it does not match the records in our identity system. In the transmission process, we use WSS (HTTPS-based), the connection information is encrypted, token will not be stolen.
Q: Will there be an SDK interface in addition to the CLI and the Web interface? This may be used by third parties.
A: The key to the client's implementation is to serialize the serialization of the message, and we are using the PROTOBUF3 definition, PROTOBUF3 can generate code in a variety of mainstream languages, so there is no need to develop the SDK separately.
The above content is organized according to the March 21, 2017 night group sharing content. Share people
Bai-Chien, the research and development engineer of Big Data Innovation Center, is mainly responsible for the development and maintenance of the open source platform lain related building, as well as the operation of the department infrastructure. Dockone Weekly will organize the technology to share, welcome interested students add: Liyingjiesz, into group participation, you want to listen to the topic or want to share the topic can give us a message.
