Python Wsgi,paste,routes,webob of OpenStack Foundation

in the reading OpenStack each component source code process, discovers all modules like Nova,neutron and so on, all through the WSGI way provides the restful API externally.

and in the process of using WSGI, it involves paste,routes,webob the use of these Python third party libraries. So if you want to understand OpenStack's code to provide restful APIs, you have to learn Python WSGI first and understand the role paste,routes,webob these libraries play.

on the internet also saw a lot of people write about this aspect of the relevant articles, found that most of them are scattered about one aspect of the concept, rarely a comprehensive introduction clearly, so decided to write a piece of this document to share, easy to read and understand OpenStack source.

all of this is around Wsgi, so we first understand the first concept Wsgi.

The WSGI is defined in PEP333:

Pep:

	333
Title:	Python Web Server Gateway Interface v1.0
Author:	Phillip J. Eby <pje at telecommunity.com>
Discussions-to:	Python Web-sig < Web-sig at python.org >
Status:	Final
Type:	Informational
Created:	07-dec-2003
Post-history:	07-dec-2003, 08-aug-2004, 20-aug-2004, 27-aug-2004, 27-sep-2010
Superseded-by:	3333

the causes and objectives of the occurrence:

Python has a large number of web framework implementations, such as Zope,quixote, Webware, Skunkweb, PSO, and Twisted web, and so on. So many frameworks are confusing for Python beginners because they think the web framework they choose will limit which Web server they use and vice versa.

as a contrast, Java also has a number of web frameworks available, but the Java "servlet API" enables Web applications written using different Web frameworks to run on different Web servers that support the Servelet API.

The purpose of WSGI is to provide an API that can be widely used in Python Web services--------------so that whether they are using Python (such as Medusa) or embedded python (such as Mod_python), or using the Cgi,fastcgi Gateway protocol to use Python applications, you can freely assemble a Python web framework or a Python Web server. Such open source Web servers and framework developers can focus on specialization in their field.

so the goal of PEP333 is to provide a simple and common interface between Web servers and Web applications: The Python Web Servergateway Interface (WSGI).

but the WSGI specification has no effect on existing Python Web servers and web frameworks, and the authors and maintainers of servers and frameworks need to implement WSGI to make the specification work.

However, since PEP333 does not have a Web server and framework to support WSGI, and there is no direct incentive to implement WSGI, the WSGI design must be easy to implement so that the investment and the cost of implementing WSGI can be very low.

This should be easy to implement for Web server and web framework side interfaces, which is critical to Wsgi, and is a key principle in any decision design.

It should be noted, however, that simplifying the implementation and framework of the framework authors is not the same as the easy use of Web applications. WSGI design interfaces do not have too much interference for framework authors, such as interference with Web response objects and cookies, can only hinder existing frameworks. So, instead of inventing a new web framework, WSGI can easily interact with the existing Web server and Web framework.

Similarly, another goal of WSGI is to be deployed in any version of Python. In this way, the new standard module is not recommended, and WSGI should not require 2.2.2 above the Python version.

In addition, to simplify the implementation of existing and future web frameworks and services, WSGI should easily create request preprocessing and response post-processing, and other WSGI based components are like an application to the servers that contain them. It also looks like a server to the application that the Web service contains.

If the middleware is simple and robust, and WSGI is widely used in Web servers and frameworks, this could lead to a new Python Web application framework: A loosely coupled framework composed of WSGI components. In addition, the current framework authors may even choose to refactor their framework in this way, so that the framework is like a library using WSGI rather than a large framework. This also allows application developers to select the most appropriate components for their applications, rather than using a single framework that has both advantages and disadvantages.

Finally, the point to be mentioned is that this PEP333 does not involve deployment. After a large number of servers and frameworks are implemented WSGI, there may be new pep to describe how to deploy the WSGI server and the application framework.

Specification Overview

The Wsgi interface includes two aspects: "Server" or "gateway" side, and application and frame side. The server side invokes a callable object provided by the application side. The server side prescribes how the object can be invoked. It can be assumed that some servers or gateways require that a deployment of the application provide a script to create a service or gateway instance, and then provide an application object to the instance. Other servers or gateways may use a configuration file or other mechanism to specify how the Application object is imported or acquired.

In addition, in order to keep servers/gateways and applications/frameworks concise, it is also possible to create middleware that implements WSGI on both sides of the specification. This middleware is like an application for servers that contain them, and for applications like the servers that contain them, middleware can be used to provide extended APIs, content transformations, navigation, and other useful functions.

in the description of the entire specification, we will use "a callable" to represent a function, a method, a class, or an instance containing a __call__ method. Choose which callable to implement depending on server,gateway or application requirements. Conversely, server,gateway, or applications cannot rely on how the callable provided to it is implemented. Callables is just being invoked.

application/Framework aspects:

The Application object is a simple callable object and accepts 2 parameters. This object term cannot be misinterpreted as requiring an actual object instance: A function, method, class, or instance with a __call__ method can be treated as an Application object. The Application object must be able to be invoked multiple times, in fact all servers/gateways (except CGI) make such a repetitive request.

Note: Although we call it an "application" object, it should not be understood to mean that the application developer uses WSGI as the programming API. It assumes that the application developers will use the current, high-level framework to develop their applications. WSGI is a framework and server developer tool that does not directly concern application developers.

Here are 2 examples of application objects, one is a function, and the other is a class.

def simple_app (environ, start_response): "" "simplest possible Application object" "status = ' OK ' response _headers = [(' Content-type ', ' Text/plain ')] start_response (status, Response_headers) return [' Hello world!\n '] C Lass Appclass: "" "produce the same output, but using a class (note: ' Appclass ' is the ' application ')  ing it returns an instance of ' Appclass ', which are then the iterable return value of the ' Application callable ' as

    required by the spec.
    If we wanted to use *instances* of ' Appclass ' as application objects instead, we would have to implement a ' __call__ '
    method, which would is invoked to execute the application, and we would need to create a instance for use by the
    Server or Gateway.

    "" "Def __init__ (self, Environ, start_response): Self.environ = environ Self.start = start_response def __iter__ (self): status = ' OK ' response_headers = [(' Content-type ', ' Text/plain ')] Self.start (status, response_headers) yield "Hello world!\n" 

Server/gateway Aspects

whenever the request is fetched from the HTTP client, the server or gateway invokes the application callable. To describe clearly, the following implements a simple CGI gateway, which uses the method as the object of application. Note that this is just a simple example of a limited error handling, because by default, an exception that is not caught will be output directly to Sys.stderr and logged by the Web server.

Import OS, sys def run_with_cgi (application): Environ = Dict (Os.environ.items ()) environ[' wsgi.input '] = Sys.stdin environ[' wsgi.errors '] = sys.stderr environ[' wsgi.version '] = (1, 0) environ[' wsgi.multit Hread '] = False environ[' wsgi.multiprocess '] = True environ[' wsgi.run_once '] = True if Environ.get (' HTTP S ', ' off ') in (' On ', ' 1 '): environ[' wsgi.url_scheme '] = ' https ' else:environ[' wsgi.url_scheme '] = ' HT TP ' Headers_set = [] Headers_sent = [] def write (data): If not headers_set:raise Asser Tionerror ("Write () before Start_response ()") Elif not headers_sent: # before the The stored headers status, Response_headers = headers_sent[:] = Headers_set sys.stdout.write (' St ATUs:%s\r\n '% status ' for header in Response_headers:sys.stdout.write ('%s:%s\r\n '% head ER) sys.stdOut.write (' \ r \ n ') sys.stdout.write (data) Sys.stdout.flush () def start_response (status, Response_head  ERs, Exc_info=none): If Exc_info:try:if headers_sent: # re-raise
                Original exception if headers sent raise exc_info[0], Exc_info[1] finally: Exc_info = None # avoid dangling circular ref elif Headers_set:raise Assertionerro

        R ("Headers already set!") headers_set[:] = [status, Response_headers] return Write result = Application (environ, start_response) tr 
        Y:for data in result:if data: # don ' t send headers until body appears write (data) If not Headers_sent:write (") # Send headers Now if body is empty finally:if hasattr (result, ' close '): Result.close ()

Middleware: Components that work in two ways

Note that for some applications, this single object may be associated with the role of the server, while other servers may be able to perform the role of the application. This middleware component provides the following features: routing request to different target application objects based on the destination URL. allows multiple applications or frameworks to execute in parallel in the same process. Network Forwarding request response to achieve load balancing and remote control. follow up on content, such as providing an XSL style sheet.

In General, the use of middleware is transparent to "server/gateway" and "application/framework" and does not require special support. Users who want to incorporate middleware into an application need to provide only one component to the server, an application, and the middleware to perform the application, as if the middleware component were server. Of course, the application of this middleware wrapper may be another application of the other middleware wrapper.

In most cases, the middleware must conform to both the Wsgi "Server/gateway" and "application/framework" Constraints and constraints. In some cases, middleware requirements are more stringent than simple servers or application, which are described in detail later.

The following is an example of a middleware that converts text/plain response to "pig Latin" by using piglatin.py. (Note: A real middleware might examine the type of content and the encoding of content in a more robust way, and this simple example ignores the possibility that words may travel through block boundaries).

From Piglatin import Piglatin class Latiniter: "" "Transform iterated output to Piglatin, if it's okay to do The "okayness" can change until the application yields its a-non-empty string, so ' TRANSFORM_OK ' has to
    Be a mutable truth value. "" "Def __init__ (self, result, TRANSFORM_OK): If hasattr (result, ' close '): Self.close = result.cl OSE Self._next = iter (Result). Next SELF.TRANSFORM_OK = Transform_ok def __iter__ (self): retur
            N Self def next (self): if Self.transform_ok:return piglatin (Self._next ()) Else: Return Self._next () class Latinator: # By default, don ' t transform output transform = False def __init_ _ (Self, application): self.application = Application def __call__ (self, Environ, start_response): TR ANSFORM_OK = [] def start_latin (status, Response_headers, Exc_info=none): # Reset OK flag, In the repeat call del transform_ok[: ' for name, value in Response_headers:
                    If name.lower () = = ' Content-type ' and value = = ' Text/plain ': Transform_ok.append (True)
                        # Strip Content-length If present, else it ' ll be wrong response_headers = [(name, value) 
                    For name, value in Response_headers if Name.lower ()!= ' content-length '

            Break write = Start_response (status, Response_headers, Exc_info)
                If Transform_ok:def write_latin (data): Write (Piglatin (data)) Return Write_latin Else:return Write return latiniter (self.application environ,  Start_latin), TRANSFORM_OK) # Run Foo_app under a latinator ' s control, using the example CGI gateway from Foo_app Import Foo_app run_with_cgi (LaTinator (Foo_app)) 

PASTE

is a handy tool kit for us to use WSGI. It provides a range of WSGI middleware that can be nested to build Web applications. Therefore, it belongs to the middleware section of the WSGI described above. All of the middleware it provides conforms to the above PEP333 interface and is compatible with other PEP333 based middleware.

It provides some of the following features:

Testing Aspects: easily test WSGI applications with Paste.fixture. test command-line applications by Paste.fixture. the Paste.lint is used to perform a static check on the component to conform to WSGI.

Dispatch: concatenation and cascading of WSGI applications via Paste.cascade (returns the first response without errors). Distribute the request to a different WSGI application via paste.urlmap based on the URL prefix. paste.recursive allows applications to create child requests and deliver requests internally.

Web applications: run the CGI program via PASTE.CGIAPP to Wsgi. Load the WSGI application from the. py file by Paste.urlparser. Load WSGI applications from the directory or egg via Paste.urlparser.

Tools: The paste.httpexceptions is used to capture HTTP-related exceptions (such as Httpnotfound) and to return the appropriate response. provides some authentication support through the Paste.auth package, including HTTP (Basic and Digest), signed cookies, and CAS single sign-on. Create a session through Paste.session and Paste.flup_session. Compress response by Paste.gzip. through Paste.request,paste.response,paste.wsgilib to provide a variety of procedures to manipulate request and produce response.

Debugging Filters: The paste.exceptions is used to catch the extension stack of the exception (using ZOPE/ZPT). Paste.cgitb_catcher to capture the error and render a CGITB based output. set up a profile for each request through Paste.debug.profile and add profiles to the HTML. paste.debug.prints is used to capture the printout and render it for debugging on the browser. through the Paste.debug.wgd_validator, verify the HTML output of the application through the WDG Validator, and add alarms or error messages to the page.

Other tools: provides file monitoring through Paste.reloader and automatically restarts the server when files are updated, such as editing code. Paste.url, a class used to generate and traverse URLs, and to create related HTML code.

mention paste, you have to mention Pastedeploy, you can see Pastedeploy as a paste expansion package, which is mainly used to discover and configure WSGI applications. For users of Wsgi, it is easy to load WSGI applications from profiles, and for WSGI developers, it is only necessary to provide a simple set of entry points for their applications. Because the middleware provided by paste conforms to the PEP333 specification, we can configure the Paste component as WSGI application when using the Pastedeploy load WSGI application. After explaining OpenStack application, you will see that it uses PASTE.URLMAP middleware.

below, use paste refer to paste and Pastedeploy, easy to explain the function and function of paste.

Let's look at how OpenStack uses paste to load its own WSGI application to neutron as an example.

first, let's look at the configuration file for the neutron API, which conforms to the Pastedeploy definition format:

/etc/neutron/api-paste.ini:

[Composite:neutron]
Use = Egg:paste#urlmap
/: Neutronversions
/v2.0:neutronapi_v2_0

[Composite:neutronapi_v2_0]
Use = Call:neutron.auth:pipeline_factory
Noauth = cors request_id catch_errors extensions Neutronapiapp_v2_0
Keystone = cors request_id catch_errors authtoken keystonecontext Extensions Neutronapiapp_v2_0

[FILTER:REQUEST_ID]
Paste.filter_factory = Oslo_middleware:RequestId.factory

[Filter:catch_errors]
Paste.filter_factory = Oslo_middleware:CatchErrors.factory

[Filter:cors]
Paste.filter_factory = Oslo_middleware.cors:filter_factory
Oslo_config_project = Neutron

[Filter:keystonecontext]
Paste.filter_factory = Neutron.auth:NeutronKeystoneContext.factory

[Filter:authtoken]
Paste.filter_factory = Keystonemiddleware.auth_token:filter_factory

[Filter:extensions]
Paste.filter_factory = Neutron.api.extensions:plugin_aware_extension_middleware_factory

[App:neutronversions]
Paste.app_factory = Neutron.api.versions:Versions.factory

[App:neutronapiapp_v2_0]
Paste.app_factory = Neutron.api.v2.router:APIRouter.factory

we will explain the application of paste by explaining the above configuration file. First, the configuration file is similar to the INI file, consisting of one configuration segment section, each of which is formatted as follows:

[Type:name]

where type includes the following: application: app,application. Filter: Filter,filte-app Pipe: Pipeline Combination: Composite

The configuration file begins with a [Composite:neutron] section, which means that it is a configuration of a combination type, and its name is neutron. The combination type indicates that it is composed of several WSGI applications. Let's look at the contents of the section in the form of key = value.

Use = Egg:paste#urlmap
/: Neutronversions
/v2.0:neutronapi_v2_0

first of all, using key ' use ' indicates that it uses the function of Paste.urlmap this middleware in the Paste egg package, which is the function of routing requests to different WSGI applications based on different URL prefixes. The following configuration indicates that:

route the access to "/" to Neutronversion, the app, and the "/v2.0" access route to the Neutronapi_v2_0 app for processing.

There are several forms of use that can be used: Egg: Objects in the egg package specified with a URI Call: Use a Callable object in a module config: Using a different configuration file

along this, let's look at the configuration of this section neutronversions :

[App:neutronversions]
Paste.app_factory = Neutron.api.versions:Versions.factory

Here's a key, "paste.app_factory." This indicates a factory function that indicates the loaded module and method.

as a validation, let's look at the implementation code for the specific WSGI application:

neutron/api/versions.py:

Class versions (object):

    @classmethod
    def factory (CLS, Global_config, **local_config): Return
        CLS (app= None)

you can see that it is true to construct the corresponding WSGI application object through the Factory factory method. This allows access to the "/" URL to be submitted to the callable Application object constructed by the factory method of version. Further, this object has a __call__ method, which is the method that is invoked when the request is sent:

@webob. Dec.wsgify (Requestclass=wsgi. Request)
def __call__ (self, req): "" "
    Respond to a request for all neutron API versions.
    " " VERSION_OBJS = [
        {
            "id": "v2.0",
            "status": "Current",
        },
    ]

This method is used to @webob.dec.wsgify this adorner, and its function is explained in detail after introducing Webob.

then look at the configuration of another neutronapi_v2_0 section:

[Composite:neutronapi_v2_0]
Use = Call:neutron.auth:pipeline_factory
Noauth = cors request_id catch_errors extensions Neutronapiapp_v2_0
Keystone = cors request_id catch_errors authtoken keystonecontext Extensions Neutronapiapp_v2_0

you can see that it is a section of the composite combination type, which is used to construct the application of the "/v2.0" URL access, first look at the first configuration:

Use = Call:neutron.auth:pipeline_factory

viewed from value "Call:neutron.auth:pipeline_factory", it is a value of the call type, stating that it uses a callable object (usually a function) to construct the corresponding Wsgi object.

neutron/auth.py:

def pipeline_factory (loader, global_conf, **local_conf): "" "
    Create a paste pipeline on the ' based ' conf IG option.
    "" " Pipeline = local_conf[cfg. Conf.auth_strategy]
    pipeline = pipeline.split ()
    filters = [Loader.get_filter (n) for n in Pipeline[:-1]]
    App = Loader.get_app (pipeline[-1])
    filters.reverse () for
    filter in filters:
        app = filter (APP)
    Return app

You can see that this method first reads the authorization policy in the configuration, there are 2 kinds of "Noauth" and "Keystone", and then reads the corresponding filters and app from the configuration file according to different authorization policies, this is a list. Because the default is "Keystone" authorization, the corresponding filters is " cors request_id catch_errors authtoken keystonecontext extensions ", and app is Neutronapiapp_v2_0, so the request for the "/v2.0" prefix URL will be returned after all filter processing is applied to the Neutronapiapp_v2_0 app. You can then see the configuration of all of the filter's apps in the configuration file:

[FILTER:REQUEST_ID]
Paste.filter_factory = Oslo_middleware:RequestId.factory

[Filter:catch_errors]
Paste.filter_factory = Oslo_middleware:CatchErrors.factory

[Filter:cors]
Paste.filter_factory = Oslo_middleware.cors:filter_factory
Oslo_config_project = Neutron

[Filter:keystonecontext]
Paste.filter_factory = Neutron.auth:NeutronKeystoneContext.factory

[Filter:authtoken]
Paste.filter_factory = Keystonemiddleware.auth_token:filter_factory

[Filter:extensions]
Paste.filter_factory = Neutron.api.extensions:plugin_aware_extension_middleware_factory

[App:neutronversions]
Paste.app_factory = Neutron.api.versions:Versions.factory

[App:neutronapiapp_v2_0]
Paste.app_factory = Neutron.api.v2.router:APIRouter.factory

the "key=value" keys in these filter and app are paste.app_factory, some pase.filter_factory, and these different key meanings are as follows: