In-depth parsing of the template engine built into Python's tornado framework

The _parse method in the template is a parser for templates grammar, and in this file a lump of various node and block is the bearer of the analytic result, that is, after the parse processing, The HTML template of the tornado we entered becomes a collection of blocks.
The ancestors of these blocks and Node are this "abstract" class, _node, which defines three method definitions, in which the Generate method must be provided by the subclass (so I call it "abstract" class).
Theoretically, when a class becomes an ancestor class, it must mean that the class contains some common behaviors in subclasses, so the method exposed by _node, that is, all subclasses theoretically have the following characteristics:
1. Can be used as a container (Each_child, Find_named_blocks)
2. Generate
Of course, the ideal is always full, there is always something wrong with the reality, for some descendants, their characteristics do not seem so reliable, such as _text.
_text This class uses only the Generate method, which is used to add text (Html, JS) to the input stream after trimming it, and if you call its Each_child or find_named_blocks, of course you can, but it doesn't make any sense.
Before repeating the _parse method, it returns the result of a _chunklist instance, while _chunklist inherits from _node. This is a class that embodies the characteristics of the _node container, overriding the Generate method and the Each_child method, and basically the related method of invoking all the elements in the container in turn.
_nodes many descendants of the more wonderful is _extendsblock this class, ya do not do anything (that is true), looks like another "abstract class", but actually will be _parse initialized, This token (tornado term) is used to process extends. I wondered, once the goods were generate, wouldn't it throw an anomaly out of the wood?
What's really interesting is that there are several other methods that have common patterns, using _applyblock to illustrate
In _applyblock, the interesting thing is the Generate method

def generate (self, writer):   method_name = "apply%d"% writer.apply_counter   writer.apply_counter + = 1   Writer.write_line ("Def%s ():"% method_name, self.line) with   writer.indent ():     writer.write_line ("_buffer = [] ", Self.line)     writer.write_line (" _append = _buffer.append ", self.line)     self.body.generate (writer)     Writer.write_line ("Return _utf8 ("). Join (_buffer) ", Self.line)   writer.write_line (" _append (%s ()) "% (     

Simply put, this function does two things:
Defines a Python file global function called APPLYXXX ():, where xxx is a shaped, self-increment value, the return value is a UTF8 string.
Execute the APPLYXXX function, and then use the output of this function as input to the Self.method function.
So, if a template like this

You will get an output similar to the following:

Tornado's template mechanism, in essence, allows developers to write view templates in HTML + template marker, but behind the scenes, Tornado will take these view templates through template processing, To become a compiled Python code.
Take Autumn-sea the above code as an example, it is easier to understand:
View Template

        {{title}}           Hello! {{Name}}     

After processing

_buffer = [] _buffer.append (' \\n\\n')  _tmp = title if Isinstance (_tmp, str): _buffer.append (_tmp) elif isinstance (_tmp, Unicode): _buffer.append (_ Tmp.encode (' utf-8 ')) Else: _buffer.append (str (_tmp))  _buffer.append ('\\n\\n\\n ') _buffer.append (' hello! ')  _tmp = name if Isinstance (_tmp, str): _buffer.append (_tmp) elif isinstance (_tmp, Unicode): _buffer.append (_ Tmp.encode (' utf-8 ')) Else: _buffer.append (str (_TMP))  

Example anatomy
Tornado template is basically in template.py this file, just over 800 lines of code to achieve the basic template available, let us slowly uncover her veil.
First, let's see how tornado compiles the template, and here's a simple template

t = Template ("" \ {%if names%}   {% for name in names%}     {{name}}   

Tornado finally compiles the following code:

Def _tt_execute (): #
 
  : 0 _tt_buffer = [] #
  
   : 0 _tt_append = _tt_buffer.append #
   
    : 0 if names: #
    
     : 1 _tt_append (' \ n ') #
     
      : 2 for name in Names: #
      
       : 2 _tt_append (' \ n ') #
       
        : 3 _tt_tmp = name #
        
         
          : 3 If Isinstance (_tt_tmp, _tt_string_types): _tt_tmp = _tt_utf8 (_tt_tmp) # 
          
           : 3 else: _tt_tmp = _tt_utf8 (str ( _TT_TMP) # 
           
            : 3 _tt_tmp = _tt_utf8 (Xhtml_escape (_tt_tmp)) # 
            
             : 3 _tt_append (_tt_tmp) # 
             
               : 3 _tt_append (' \ n ') # 
              
               : 4 Pass # 
               
                : 2 _tt_append (' \ n ') # 
                
                 : 5 Pass # 
                 
                  : 5 Else: # 
                  
                   : 5 _tt_append (' \nno one\n ') # 
                   
                    : 7 p # 
                    
                     : 1 _tt_append (' \ n ') # 
                     
                      : 8 return _tt_utf8 ("). Join (_tt_ Buffer) # 
                      
                       : 0
                       
                      
                     
                    
                   
                  
                 
                
               
              
             
            
           
         
        
       
      
     
    
   
  
 

Yes, you're right, tornado compile is to translate it into blocks of code, and finally exec passes the parameter namespace we give to execute the _tt_execute function. The
includes 4 predefined node nodes in the template above, each of which has its own build _controlblock,_expression,_text.
For example, the _expression expression node, which is the {{name}} in our template, when _parse resolves to find that ' {' followed by ' {' is considered an expression node,

Class _expression (_node):   def __init__ (self, expression, line, Raw=false):     self.expression = Expression     Self.line = line     Self.raw = raw    def generate (self, writer):     writer.write_line ("_tt_tmp =%s"% Self.expression, Self.line)     writer.write_line ("If Isinstance (_tt_tmp, _tt_string_types):"              "_tt_tmp = _tt_ UTF8 (_tt_tmp) ", Self.line)     writer.write_line (" else: _tt_tmp = _tt_utf8 (str (_tt_tmp)) ", self.line)     if not Self.raw and Writer.current_template.autoescape is not the None:       # in Python3 functions like Xhtml_escape return Unicode,       # So we had to convert to utf8 again.       Writer.write_line ("_tt_tmp = _tt_utf8 (%s (_tt_tmp))"%                Writer.current_template.autoescape, self.line)     

The Generate method of the node is called at the end of the build, and Self.expression is the name above, so the value of name is append to the internal list when exec.
Like If,for and so on are control nodes, they are defined as follows:

Class _controlblock (_node):   def __init__ (self, statement, line, Body=none):     self.statement = statement     Self.line = line     self.body = Body    def each_child (self):     return (Self.body,)    def generate (self, writer) :     writer.write_line ("%s:"% self.statement, self.line) with     writer.indent ():       Self.body.generate (writer )       # Just in case the body is empty       

The Generate method of the control node is a bit of a point, because if,for and so on is the next line is required to indent, so called with writer.indent continue indentation control, you can look at the
The indent method of _codewriter.
The interesting point in the node is _extendsblock, which is the node that implements the target base,

Class _extendsblock (_node):   def __init__ (self, name):     

We find that the Generate method is not defined, and will not be an error when generating an inheritance node? Let's take a look at some examples

The current directory is base.html as follows:

    {{title}}      {% Block login_name%}hello! {{Name}} {% END%}     

We can look at the parsed node,

Since we inherited the base.html, we should generate it in base.html template and replace block in base.html with the newly defined block.
This is a very normal way of thinking, Tornado is really doing so, but the deal is not in the _extendsblock.
And the _generate_python of the real template

def _generate_python (self, loader, compress_whitespace):    buffer = Stringio ()    try:      # Named_blocks maps from Names to _namedblock objects      named_blocks = {}      ancestors = self._get_ancestors (loader)      Ancestors.reverse ( ) for      ancestor in ancestors:        ancestor.find_named_blocks (loader, named_blocks)      writer = _codewriter ( Buffer, named_blocks, loader, ancestors[0].template,                compress_whitespace)      Ancestors[0].generate (writer)      return Buffer.getvalue ()    finally:      Buffer.close ()   def _get_ancestors (self, loader):    Ancestors = [Self.file] for    Chunk in self.file.body.chunks:      if Isinstance (Chunk, _extendsblock):        if not Loader:          Raise ParseError ("{% extends%} block found, but no"                  "template loader")        template = Loader.load (ch Unk.name, Self.name)        ancestors.extend (template._get_ancestors (loader))    

Call _get_ancestors in _generate_python to get the parent template of the current template, we see if the current template's _file node has _ Extendsblock represents having a parent template and loading the parent template through Loader.load, when the parent template is already a parsed _file node. So, in the template above, ancestors is [the current template _file node, the parent template _file node],ancestors.reverse () actually ancestors[0] is the parent template, we see finally through the ancestors[0]. Generate (writer) to generate the code. How does the current template replace the block content of the parent template?
See, Block login_name by parsing to _namedblock, in _generate_python by calling Ancestor.find_named_blocks to replace
The _namedblock of the parent template.

For ancestor in ancestors:    ancestor.find_named_blocks (loader, named_blocks) ancestor is actually _file node, find_named_ Blocks will traverse all nodes in the _file node and call Find_named_blocks class _namedblock (_node):   def find_named_blocks (self, loader, named_ Blocks):     named_blocks[self.name] = self     

Other nodes find_named_blocks do nothing, _namedblock by named_blocks[self.name] = self to replace the _namedblock of the current template, because the ancestors parent template is in front, The current template is behind, so the last one used is the _namedblock of the current template.
After generating the code, generate will exec the code in the given namespace

def generate (self, **kwargs): "" "Generate this template with the given arguments." " Namespace = {"Escape": Escape.xhtml_escape, "Xhtml_escape": Escape.xhtml_escape, "Url_escape": Escape.url_esc Ape, "Json_encode": Escape.json_encode, "Squeeze": Escape.squeeze, "linkify": Escape.linkify, "datetime": datetime, "_tt_utf8": Escape.utf8, # for internal use "_tt_string_types": (Unicode_type, Bytes_type), # __name     __ and __loader__ allow the traceback mechanism to find # The generated source code. "__name__": Self.name.replace ('. ', ' _ '), "__loader__": Objectdict (Get_source=lambda name:self.code),} namespace.    Update (Self.namespace) namespace.update (Kwargs) exec_in (self.compiled, namespace) Execute = namespace["_tt_execute"] # Clear The Traceback module ' s cache of source data now that # we ' ve generated a new template (mainly for this module   ' s # unittests, where different tests reuse the same name). Linecache.clearcache ()   Return Execute ()  

So in the template can use DateTime, etc., are injected into the template here, of course, there are others through
Get_template_namespace injected in the web.py

def get_template_namespace (self): "" "    Returns a dictionary to be used as the default Template namespace.     May is overridden by subclasses to add or modify values.     The results of this method is combined with additional defaults in the    ' tornado.template ' module and keyword Argu ments to    ' render ' or ' render_string '.    "" " namespace = Dict (      handler=self,      request=self.request,      current_user=self.current_user,      locale= Self.locale,      _=self.locale.translate,      static_url=self.static_url,      xsrf_form_html=self.xsrf_form _html,      reverse_url=self.reverse_url    )    namespace.update (Self.ui)    

Let's look at how Tornado's templates support the UI module.

{% for entry in entries%}  

The _module node is generated when the module is used

Class _module (_expression):   def __init__ (self, Expression, line):     super (_module, self). __init__ ("_tt_ Modules. "+ expression, line,                    

We see that the _module node is actually inherited from the _expression node, so the last thing to do is _tt_modules. Entry (Entry)
_tt_modules defined in the RequestHandler of web.py

self.ui["_tt_modules"] = _uimodulenamespace (self,application.ui_modules)

and injected into the template through the get_template_namespace above.

Class _uimodulenamespace (object): "" "   Lazy namespace which creates Uimodule proxies bound to a handler.   " " def __init__ (self, Handler, ui_modules):     Self.handler = handler     self.ui_modules = Ui_modules    def __getitem __ (self, key):     return Self.handler._ui_module (Key, Self.ui_modules[key])    def __getattr__ (self, key):     Try:       return Self[key]     except Keyerror as E:       

So when executing _tt_modules. Entry (Entry) first accesses the __getattr__ of _uimodulenamespace, then accesses __getitem__, and finally calls
Handler._ui_module (Key, Self.ui_modules[key]),

def _ui_module (self, Name, module):   def render (*args, **kwargs):     if isn't hasattr (self, "_active_modules"):       Self._active_modules = {}     if name not in Self._active_modules:       self._active_modules[name] = module (self)     rendered = Self._active_modules[name].render (*args, **kwargs)     return rendered   

_tt_modules. Entry (Entry) Entry will be passed to the _ui_module inside the render, that is Args=entry
Self._active_modules[name] = module (self) is now the instantiated uimodule, calling render to get the rendered content

Class Entry (Tornado.web.UIModule):   def render (self, Entry, show_comments=false):     return self.render_string (       

Of course, if you think this is a bother, you can also use Tornado's own Templatemodule, which inherits from Uimodule,
You can use it like that.

You can reference the required static files in module_entry.html by set_resources

It is important to note that the Set_resources function can only be used in the HTML file referenced by the template, because set_resources is the intrinsic function of Templatemodule.render

Class Templatemodule (Uimodule): "" "uimodule that simply renders the given template. {% Module Template ("foo.html")%} is similar to {% = "foo.html"%}, but the module version gets its own namespace    (with Kwargs passed to Template ()) instead of inheriting the outer Template ' s namespace. Templates rendered through this module also get access to Uimodule ' s automatic javascript/css features. Simply call set_resources inside the template and give it keyword arguments corresponding to the methods on Uimodule:   {{set_resources (Js_files=static_url ("My.js")}} Note that these resources is output once per template file, not once per instantiation of the template, so they must no   T depend on any arguments to the template. "" "Def __init__ (self, Handler): Super (Templatemodule, self). __init__ (handler) # Keep resources in both a list a   nd a dict to preserve order self._resource_list = [] Self._resource_dict = {} def render (self, Path, **kwargs):   def set_resources (**kwargs): If Path not in Self._resource_dict:self._resource_list.append (Kwargs) Self._resource_dict[path] = Kwargs else:if Self._resource_dict[path]! = kwargs:raise Valueerr  or ("Set_resources called with Different", "resources for the same template") return "" Return  Self.render_string (Path, set_resources=set_resources, **kwargs)

