In-depth analysis of Python's Tornado framework's built-in template engine

The template engine is the key for front-end presentation in the Web development framework. here we will use examples to thoroughly parse the built-in template engine in the Python Tornado framework to learn how to compile Tonardo templates. the _ parse method in the template is the parser of the template syntax, and a bunch of nodes and blocks in this file are the carriers of the parsing results, that is, after parse processing, the html template of tornado we entered becomes a collection of various blocks.
These blocks and nodes are the "abstract" class, _ node, which defines three methods, the generate method must be implemented by the subclass (so I call it an "abstract" class ).
Theoretically, when a class becomes an ancestor class, it must mean that this class contains some common behavior in the subclass. so, from the method exposed by _ Node, in theory, all subclasses have the following features:
1. it can be used as a container (each_child, find_named_blocks)
2. generate
Of course, the ideal is always full, and there is always something wrong in reality. for some children, their features are not so reliable, such as _ Text.
The _ Text class only uses the generate method, which is used to add Text (Html, JS) to the input stream after being trim. if you call its each_child or find_named_blocks, of course you can do this, but it makes no sense.
As mentioned above, the _ Parse method returns an instance of _ ChunkList, while _ ChunkList inherits from _ Node. This is a class that reflects the characteristics of the _ Node container. it overwrites the generate method and the each_child method, and basically calls the related methods of all elements in the container in sequence.
Among the many sons and grandchildren of _ Nodes, the odd one is the _ ExtendsBlock class. it seems like another "abstract class" That does nothing (That is true ", but it will be initialized by _ Parse to process the token (tornado term) of Extends ). I am wondering, once this product is generate, won't it throw an exception?
What is really interesting is that there are several other methods that share the same pattern. use _ ApplyBlock for example.
In _ ApplyBlock, the interesting thing is that the generate method

def generate(self, writer):   method_name = "apply%d" % writer.apply_counter   writer.apply_counter += 1   writer.write_line("def %s():" % method_name, self.line)   with writer.indent():     writer.write_line("_buffer = []", self.line)     writer.write_line("_append = _buffer.append", self.line)     self.body.generate(writer)     writer.write_line("return _utf8('').join(_buffer)", self.line)   writer.write_line("_append(%s(%s()))" % (     self.method, method_name), self.line) 

In short, this function does two things:
Defines a python file global function called applyXXX (): where XXX is an integer, an auto-increment value, and the return value is a utf8 string.
Execute the applyXXX function and use the output of this function as the input of the function self. method.
Therefore, if a template like this

{%apply linkify%} {{address}} {%end%} 

The output is similar to the following:

r = applyXXX() r = linkify(r) _append(r) 

In essence, tornado's template mechanism allows developers to write View templates in HTML + template marker mode. However, tornado will process these View templates through templates, to compile python code.
Taking the code above autumn-sea as an example, it is easier to understand:
View Template

        {{ title }}           hello! {{ name }}     

After processing

_buffer = [] _buffer.append('\\n\\n')  _tmp = title if isinstance(_tmp, str): _buffer.append(_tmp) elif isinstance(_tmp, unicode): _buffer.append(_tmp.encode('utf-8')) else: _buffer.append(str(_tmp))  _buffer.append('\\n\\n\\n') _buffer.append('hello! ')  _tmp = name if isinstance(_tmp, str): _buffer.append(_tmp) elif isinstance(_tmp, unicode): _buffer.append(_tmp.encode('utf-8')) else: _buffer.append(str(_tmp))  _buffer.append('\\n\\n\\n') return ''.join(_buffer)\n" 

Instance profiling
Tornado templates are basically in the template. py file. in just over 800 lines of code, the basic templates are available. let's unveil them slowly.
First, let's take a look at how tornado compiles the template. Below is a simple template.

t = Template("""\ {%if names%}   {% for name in names %}     {{name}}   {%end%} {%else%} no one {%end%} """) 

The final compilation code of tornado is as follows:

def _tt_execute(): # 
 
  :0   _tt_buffer = [] # 
  
   :0   _tt_append = _tt_buffer.append # 
   
    :0   if names: # 
    
     :1     _tt_append('\n  ') # 
     
      :2     for name in names: # 
      
       :2 _tt_append('\n ') # 
       
        :3 _tt_tmp = name # 
        
         :3 if isinstance(_tt_tmp, _tt_string_types): _tt_tmp = _tt_utf8(_tt_tmp) # 
         
          :3 else: _tt_tmp = _tt_utf8(str(_tt_tmp)) # 
          
           :3 _tt_tmp = _tt_utf8(xhtml_escape(_tt_tmp)) # 
           
            :3 _tt_append(_tt_tmp) # 
            
             :3 _tt_append('\n ') # 
             
              :4 pass # 
              
               :2 _tt_append('\n') # 
               
                :5 pass # 
                
                 :5 else: # 
                 
                  :5 _tt_append('\nno one\n') # 
                  
                   :7 pass # 
                   
                    :1 _tt_append('\n') # 
                    
                     :8 return _tt_utf8('').join(_tt_buffer) # 
                     
                      :0 
                     
                    
                   
                  
                 
                
               
              
             
            
           
          
         
        
       
      
     
    
   
  
 

Yes, you are not mistaken. tornado compilation is to translate it into code blocks, and finally execute the _ tt_execute function through the parameter namespace passed to us through exec.
The preceding template contains four predefined NODE types: _ ControlBlock, _ Expression, and _ TEXT. each Node has its own generation method.
For example, the _ Expression node, that is, the {name} in our template, is considered an Expression node when '{' is followed by '{' during _ parse parsing,

class _Expression(_Node):   def __init__(self, expression, line, raw=False):     self.expression = expression     self.line = line     self.raw = raw    def generate(self, writer):     writer.write_line("_tt_tmp = %s" % self.expression, self.line)     writer.write_line("if isinstance(_tt_tmp, _tt_string_types):"              " _tt_tmp = _tt_utf8(_tt_tmp)", self.line)     writer.write_line("else: _tt_tmp = _tt_utf8(str(_tt_tmp))", self.line)     if not self.raw and writer.current_template.autoescape is not None:       # In python3 functions like xhtml_escape return unicode,       # so we have to convert to utf8 again.       writer.write_line("_tt_tmp = _tt_utf8(%s(_tt_tmp))" %                writer.current_template.autoescape, self.line)     writer.write_line("_tt_append(_tt_tmp)", self.line) 

The generate method of the node will be called at the end of generation. self. expression is the above name, so the value of name will be appended to the internal list during exec.
For example, if and for are all control nodes. their definitions are as follows:

class _ControlBlock(_Node):   def __init__(self, statement, line, body=None):     self.statement = statement     self.line = line     self.body = body    def each_child(self):     return (self.body,)    def generate(self, writer):     writer.write_line("%s:" % self.statement, self.line)     with writer.indent():       self.body.generate(writer)       # Just in case the body was empty       writer.write_line("pass", self.line) 

The generate method of the control node is a bit meaningful, because if, for and so on are the next line that needs to be indented, so with writer. indent is called to continue indenting control. you can see below
_ CodeWriter indent method.
_ ExtendsBlock is interesting in the node, which is the basic node for achieving the target,

class _ExtendsBlock(_Node):   def __init__(self, name):     self.name = name 

We found that the generate method is not defined. isn't an error reported when an inherited node is generated? Let's look at an example.

loader = Loader('.') t=Template("""\ {% extends base.html %} {% block login_name %}hello world! {{ name }}{% end %} """,loader=loader) 

Base.html under the current directory is as follows:

    {{ title }}      {% block login_name %}hello! {{ name }}{% end %}     

We can look at the parsed nodes,

Since we inherited base.html, we should generate it with the template of base.html and use the newly defined block to replace the block in base.html,
This is a normal idea. tornado does the same thing, but it does not deal with _ ExtendsBlock.
In _ generate_python of the actual Template

def _generate_python(self, loader, compress_whitespace):    buffer = StringIO()    try:      # named_blocks maps from names to _NamedBlock objects      named_blocks = {}      ancestors = self._get_ancestors(loader)      ancestors.reverse()      for ancestor in ancestors:        ancestor.find_named_blocks(loader, named_blocks)      writer = _CodeWriter(buffer, named_blocks, loader, ancestors[0].template,                compress_whitespace)      ancestors[0].generate(writer)      return buffer.getvalue()    finally:      buffer.close()   def _get_ancestors(self, loader):    ancestors = [self.file]    for chunk in self.file.body.chunks:      if isinstance(chunk, _ExtendsBlock):        if not loader:          raise ParseError("{% extends %} block found, but no "                  "template loader")        template = loader.load(chunk.name, self.name)        ancestors.extend(template._get_ancestors(loader))    return ancestors 

In _ generate_python, call _ get_ancestors to obtain the parent template of the current template. we can see that if _ ExtendsBlock exists in the _ FILE node of the current template, it indicates that there is a parent template and loader. load the parent template. at this time, the parent template is already a parsed _ FILE node. Therefore, in the preceding template, ancestors is [current template _ FILE node, parent template _ FILE node], and ancestors. after reverse (), ancestors [0] is actually the parent template. we can see that ancestors [0] is used at last. generate (writer) to generate code. Then how does the current template replace the block content of the parent template?
Check that block login_name is resolved to _ NamedBlock, and replaced by calling ancestor. find_named_blocks in _ generate_python.
The _ NamedBlock of the parent template.

For ancestor in ancestors: ancestor. handler (loader, named_blocks) ancestor is actually a _ FILE Node. find_named_blocks will traverse all nodes in the _ FILE Node and call find_named_blocks class _ NamedBlock (_ Node): def find_named_blocks (self, loader, named_blocks): named_blocks [self. name] = self _ Node. find_named_blocks (self, loader, named_blocks)

Other node find_named_blocks does not do anything. _ NamedBlock uses named_blocks [self. name] = replace self with the _ NamedBlock of the current template. because the ancestors parent template is in the front and the current template is in the back, the _ NamedBlock of the current template is used at the end.
After the code is generated, generate will execute the code in the given namespace.

def generate(self, **kwargs):   """Generate this template with the given arguments."""   namespace = {     "escape": escape.xhtml_escape,     "xhtml_escape": escape.xhtml_escape,     "url_escape": escape.url_escape,     "json_encode": escape.json_encode,     "squeeze": escape.squeeze,     "linkify": escape.linkify,     "datetime": datetime,     "_tt_utf8": escape.utf8, # for internal use     "_tt_string_types": (unicode_type, bytes_type),     # __name__ and __loader__ allow the traceback mechanism to find     # the generated source code.     "__name__": self.name.replace('.', '_'),     "__loader__": ObjectDict(get_source=lambda name: self.code),   }   namespace.update(self.namespace)   namespace.update(kwargs)   exec_in(self.compiled, namespace)   execute = namespace["_tt_execute"]   # Clear the traceback module's cache of source data now that   # we've generated a new template (mainly for this module's   # unittests, where different tests reuse the same name).   linecache.clearcache()   return execute() 

Therefore, you can use datetime in the template by injecting it to the template here. of course, there are other
Get_template_namespace injected in web. py

 def get_template_namespace(self):    """Returns a dictionary to be used as the default template namespace.     May be overridden by subclasses to add or modify values.     The results of this method will be combined with additional    defaults in the `tornado.template` module and keyword arguments    to `render` or `render_string`.    """    namespace = dict(      handler=self,      request=self.request,      current_user=self.current_user,      locale=self.locale,      _=self.locale.translate,      static_url=self.static_url,      xsrf_form_html=self.xsrf_form_html,      reverse_url=self.reverse_url    )    namespace.update(self.ui)    return namespace 

Let's take a look at how tornado templates support the UI module.

{% for entry in entries %}  {% module Entry(entry) %} {% end %} 

When a module is used, _ Module node is generated.

class _Module(_Expression):   def __init__(self, expression, line):     super(_Module, self).__init__("_tt_modules." + expression, line,                    raw=True) 

We can see that the _ Module node is inherited from the _ Expression node, so the last execution is _ tt_modules.Entry (entry)
_ Tt_modules is defined in RequestHandler of web. py.

self.ui["_tt_modules"] = _UIModuleNamespace(self,application.ui_modules)

And use the preceding get_template_namespace to add to the template.

class _UIModuleNamespace(object):   """Lazy namespace which creates UIModule proxies bound to a handler."""   def __init__(self, handler, ui_modules):     self.handler = handler     self.ui_modules = ui_modules    def __getitem__(self, key):     return self.handler._ui_module(key, self.ui_modules[key])    def __getattr__(self, key):     try:       return self[key]     except KeyError as e:       raise AttributeError(str(e)) 

Therefore, when _ tt_modules.Entry (entry) is executed, access _ getattr __of _ UIModuleNamespace, access _ getitem __, and call
Handler. _ ui_module (key, self. ui_modules [key]),

def _ui_module(self, name, module):   def render(*args, **kwargs):     if not hasattr(self, "_active_modules"):       self._active_modules = {}     if name not in self._active_modules:       self._active_modules[name] = module(self)     rendered = self._active_modules[name].render(*args, **kwargs)     return rendered   return render 

The entry in _ tt_modules.Entry (entry) will be passed to the render inside _ ui_module, that is, args = entry
Self. _ active_modules [name] = module (self) is the instantiated UIModule, which calls render to obtain the rendered content.

class Entry(tornado.web.UIModule):   def render(self, entry, show_comments=False):     return self.render_string(       "module-entry.html", entry=entry, show_comments=show_comments) 

If you think this is troublesome, you can also use the TemplateModule that comes with tornado, which inherits from the UIModule,
You can use this

{% module Template("module-entry.html", show_comments=True) %} 

In module_entry.html, you can use set_resources to reference the required static file.

{{ set_resources(embedded_css=".entry { margin-bottom: 1em; }") }} 

Note that the set_resources function can only be used in the html file referenced by Template, because set_resources is the internal function of TemplateModule. render.

class TemplateModule(UIModule):   """UIModule that simply renders the given template.    {% module Template("foo.html") %} is similar to {% include "foo.html" %},   but the module version gets its own namespace (with kwargs passed to   Template()) instead of inheriting the outer template's namespace.    Templates rendered through this module also get access to UIModule's   automatic javascript/css features. Simply call set_resources   inside the template and give it keyword arguments corresponding to   the methods on UIModule: {{ set_resources(js_files=static_url("my.js?1.1.9")) }}   Note that these resources are output once per template file, not once   per instantiation of the template, so they must not depend on   any arguments to the template.   """   def __init__(self, handler):     super(TemplateModule, self).__init__(handler)     # keep resources in both a list and a dict to preserve order     self._resource_list = []     self._resource_dict = {}    def render(self, path, **kwargs):     def set_resources(**kwargs):       if path not in self._resource_dict:         self._resource_list.append(kwargs)         self._resource_dict[path] = kwargs       else:         if self._resource_dict[path] != kwargs:           raise ValueError("set_resources called with different "                    "resources for the same template")       return ""     return self.render_string(path, set_resources=set_resources,                  **kwargs)