Swift START process & Data Flow

Speaking of the source code reading of large systems, the swift currently in progress has only two experiences (I have read and learned the openfire source code in the first half of last year ). Although it is still a cainiao level, we can also sum up the experience twice: P, haha ~

In my experience, it is best for the source code of this server.First, we will track and read the system's "Startup Process" and "data flow" when the request arrives., UnderstandProgramAs well as the relationship between various key classes and methods,Then, we will learn from the main thread in detail about each branch process.. This method is more purposeful,CodeIt's easy to see. Second, it's easy to see more details.

Okay, let's get started. In the past two weeks, I started learning about SWIFT source code (SWIFT 1.8.0). First, I have read it over the process described above. Now I will summarize it.

 

Swift START process ==================================================== ================================

After deploying Swift on Ubuntu, we know that the script can be executed.StartmainWhen swift is started, how does swift start after this command is executed?

1. Overview

StartmainThe script actually runs the following command:

 
Swift-init main start
 

That is, the initialization is executed for swift, and the Command actually runsSwift-proxy-server, swift-account-server, swift-container-server, swift-object-ServerThese four Python commands in swift bin. Next, let's take a look at what has been done in Swift code.

2. Swift-init

Swift-initIs the script to start Swift. The script can be changed with several parameters:Swift-init [subject] [action]. In theSwift-init main start", The main corresponds to the subject, and the action is start.

1)Swift-init main startAfter the code is executed, first check the swift-init code line61 ~ Line62: the commands and parameters are separated here to determine the next task:

 
Command = ARGs [-1]#Command: Start.Servers = ARGs [:-1]#Servers here is main
 

 

2) InSwift-init code line 70 creates a manager instance, which directly manages various servers:

 
Manager = Manager (servers, run_dir = options. run_dir)#Here, the manager's _ init _ is executed __
 

 

3) as swift-init created the manager instance in the previous step, the _ init _ Initialization Method of the manager is called. In the _ init _ method, we can see the processing of the command parameter servers (here is "Main") (line 137 ).

 
ElifServer ='Main': Server_names.update (main_servers)
 

Main_servers is the Global Array defined in manager. py, indicating the start script (line 37) of servers in swift)

Main_servers = ['Proxy-Server','Account-Server','Container-Server','Object-Server']
 

When manager. _ init _ finally adds servers, four servers in swift are created: Swift-proxy-server, swift-account-server, swift-container-server, swift-object-Server 4) after the manager instance is created, return to swift-init. At line 72, the run_command method of the manager is called and the start command is obtained:

 
Status = manager. run_command (command, ** options._ Dict __)
 

5) in the Manager. in the run_cmmand method (line 162) of Py, the start method is finally executed, that is, the _ main __of swift-proxy-server, swift-account-server, swift-container-server, and swift-object-server are executed __, that is, the launch method called in the Start method prints this heap:

Here, the various servers in swift are started. Let's take proxy-server as an example to see what has been done in these swift-XX-server.

 

3.Swift-XX-server startup

In swift-XX-server, the _ init _ of the corresponding class in XX. server. py is finally called (take proxy as an example ):

 
Run_wsgi (conf_file,'Proxy-Server', Default_port = 8080, ** options)
 

We can see that SWIFT's HTTP interaction is based onWsgiAnd it usesPaste deployManage the configuration file to load the corresponding wsgi app through the xx-server.conf configuration file. In the run_wsgi method, perform the following four steps:

1) The run_wsgi method loads the configuration file according to the parameter

2) create a socket based on the port bound to the configuration file and bind the socket to the server.

3) The run_server method calls the _ init _ method of the proxy server.

4) The _ init _ method initializes the internal status of the server according to the configuration file, so as to gets the server ready to handle requests

 

So far, each servers in SWIFT has been stand! Only waiting for the arrival of request! OK, so let's take a look at the data flow in SWIFT when a request arrives :)

 

 

SWIFT data flow========================================================== ========================

As mentioned above, Swift's HTTP interaction framework is implemented based on wsgi, and paste deploy is used to load the app through Conf, therefore, if you want to better understand the swift interaction process and the wsgi-style application standard, we recommend that you first learn the wsgi standard and the meaning of the paste deploy configuration file.

First, let's take a look at a picture! This figure shows the data flow of the entire request process! (Forgive me for playing the watermark shamelessly, haha)

Let's explain it according to the process shown in the figure.

1. Send a request

First, the user, the client, sends a request to SWIFT, which enters SWIFT through the bind_port configured in the proxy-server.conf;

 

2. wsgi middleware: Pipeline

Here we first give a proxy-server.conf configuration content, we will combine this paste deploy configuration file for pipeline instructions:

 [Default] bind_port = 8080 User = Rootlog_facility = Log_local1eventlet_debug = True[Pipeline: Main] Pipeline  = Healthcheck cache tempauth proxy-loggingProxy-  Server[App: proxy -Server] Use = Egg: swift #  Proxy Allow_account_management = Trueaccount_autocreate = True [filter: tempauth] Use = Egg: swift #  Tempauth User_admin_admin = Admin. admin. reseller_adminuser_test_tester = Testing. adminuser_test2_tester2 = Testing2. adminuser_test_tester3 = Testing3 [filter: healthcheck] Use = Egg: swift#  Healthcheck  [Filter: cache] Use = Egg: swift #  Memcache  [Filter: proxy - Logging] Use = Egg: swift #  Proxy_logging 

The request must pass through layer-by-layer filters in wsgi pipeline to reach the requested application: proxy-server. Note the bold Section in the configuration file:

[Pipeline: Main] indicates that this is the application entry. Pipeline indicates the filters of some columns. Main is similarProgramming LanguageThe main method in.

Pipeline = filter1 filter2... the last element of the filtern application must be a wsgi-compliant callable object. In this example, the request must be filtered by the healthcheck, cache tempauth, and proxy-logging filters, to the proxy-server. If the request does not pass any filter, the request will be returned with an error message and cannot reach the proxy-server.

These filters and the final app must both be wsgi-compliant callable applications. During program execution, their _ call _ methods are called in sequence. Let's take healthcheck as an example, let's take a look at its _ call __:

 Def   _ Call __  (Self, ENV, start_response): req = Request (ENV)  Try  :  If Req. Path = '  /Healthcheck '  : Handler = Self. Get  If Self. disable_path And  OS. Path. exists (self. disable_path): Handler = Self. Disabled  Return  Handler (req) (ENV, start_response)  Except  Unicodeerror:  #  Definitely, this is not/healthcheck             Pass          Return Self. app (ENV, start_response)

We can see that this method must contain two parameters (here is the method in the class, so there is also a self) ENV and start_response, ENV is the environment variable dictionary in wsgi, start_response is used to generate the response result. Self. the app (ENV, start_response) is actually its next filter: cache, And the cache will be executed in the same way until the proxy-server is called, the proxy-server is responsible for generating the corresponding response and returning it.

 

3. Proxy-server. _ call __

Assuming that the request sent by the client is very legal, he has passed all the filters and now he has finally reached the proxy-server. What will happen next?

The proxy-server application actually corresponds to the application class in server. py under the swift source code proxy package. As we mentioned above, application. _ call _ will be called here __. In application. _ call _, the request is verified. After the request is valid, the handle_request method of the request is called. As the name suggests, the request is handle.

We know that requests in Swift can target accounts, iner, and objects. Obviously, their controllers are different. Therefore, the get_controller method is still called in the handle_request method.

According to the http: // xxxx/Account [container/object]/Xxx sub-path to determine the part of the request to be requested, and then obtain the corresponding controller (accountcontroller, containercontroller, objectcontroller ). Carefully check the account package, container package, and server. py in the OBJ package in the swift source code. Correct, accountcontroller, containercontroller, and objectcontroller are located in these server. py respectively!

 

4. Controllers

Now, we have determined the Controller corresponding to this request! So what is its request action? Head? Get? In fact, all this has been defined in HTTP verb. Therefore, the handle_request method will continue to call the corresponding XXX method (such as head/Put/get/post/delete) in the Controller instance ), these methods finally process the request, generate a response, and return the corresponding request!

 

5. Return request

Finally, the request is returned to the Client layer by layer. What Should swift do? Wait for its requests from the four-side eight-way method.

 

So far, the process of comparing outline requests is completed. I omitted a lot of details and verification work here, just to let everyone know, just like me, how deep it is before diving into swift source code, what is the route like. As for the details and precision of each step, we also need to enrich and learn step by step.

 

 

 

========================================= Full-text splitting ============================================

 

This article is written in the context of panic. I am confused about Python because I have learned it in many places (such as wsgi and pastedeploy). In addition, this is the first rough understanding of swift source code, so it is inevitable that there are some errors. If you have any ideas, please talk about them more. Diving = d