Deep parsing of the template engine built into the Python tornado framework _python

The _parse method in the

Template is the parser for the template grammar, and a lump of node and block in this file is the bearer of the analytic result, that is, after the parse process, The Tornado HTML template we entered becomes a collection of various blocks.
The ancestors of these blocks and Node are the "abstract" class, _node, which defines three method definitions, where the Generate method must be implemented by subclasses (so I call it the "abstract" class).
  Theoretically, when a class becomes an ancestor class, it must mean that the class contains some common behavior in subclasses, so, from the way _node exposes, all subclasses theoretically have the following characteristics:
1. Can be used as a container (Each_child, Find_named_blocks)
2. Generate
Of course, the ideal is always plump, the reality is always something wrong, for some children, their characteristics do not look so reliable, such as _text.
 _text This class uses only the Generate method, which is used to add text (Html, JS) to the input stream after being trim, and if you call it Each_child or find_named_blocks, of course you can. But it doesn't make any sense.
  Before repeating the _parse method, it returns the result of a _chunklist instance, while _chunklist inherits and _node. This is a class that embodies the characteristics of the _node container, rewriting the Generate method and the Each_child method, and basically the related method of invoking all the elements in the container in turn.
 _nodes Many of our descendants are _extendsblock this class, you do not do anything (which is true), it looks like another "abstract class", but it will be initialized by _parse, Used to process extends this token (tornado term). I wonder, once the goods are generate, do not throw an abnormal out of the wood?
  What's really interesting is that there are several other methods that have a common pattern, with _applyblock for example
  in _applyblock, interestingly generate method

def generate (self, writer): 
  method_name = "apply%d"% writer.apply_counter 
  Writer.apply_counter + 1 
  Writer.write_line ("Def%s ():"% method_name, self.line) with 
  writer.indent (): 
    writer.write_line ("_buffer = [] ", Self.line) 
    writer.write_line (" _append = _buffer.append ", self.line) 
    self.body.generate (writer) 
    Writer.write_line ("Return _utf8 ("). Join (_buffer), self.line) 
  writer.write_line ("_append (%s (%s))"% ( 
    Self.method, Method_name), Self.line) 

In simple terms, this function does two things:
Defines a Python file global function called APPLYXXX ():, where the xxx is an integer, the value of the increment, the return value is a UTF8 string.
Execute the APPLYXXX function and use the output of this function as input to the Self.method function.
So, if a template like this

{%apply linkify%} {Address}} {%end%} 

Will get an output similar to the following:

R = applyxxx () 
R = linkify (r) _append (r) 
 

Tornado's template mechanism, essentially, is to allow developers to write view templates in HTML + template marker, but in the back, Tornado will take these view templates through the template process, To become a compiled Python code.
Take Autumn-sea the above code as an example, it is easier to understand:
View Template

After processing

_buffer = [] 
_buffer.append ('  
 

Case analysis
tornado templates are basically in template.py this file, just over 800 lines of code to achieve the basic available template, let us slowly uncover her veil.
First, let's look at how tornado compiles the template, and here's a simple template

t = Template ("" \ 
{%if names%} 
  {% for name in names%} { 
    {name}} { 
  %end%} { 
%else%} 
no one< c23/>{%end%} 
"" ") 

The

Tornado finally compiles the code as follows:  

 Def _tt_execute (): # <string>:0 _tt_buffer = [] # <string>:0 _tt_append = _tt_bu  Ffer.append # <string>:0 If names: # <string>:1 _tt_append (' \ n ') # <string>:2 for name in 
      Names: # <string>:2 _tt_append (' \ n ') # <string>:3 _tt_tmp = name # <string>:3 If Isinstance (_tt_tmp, _tt_string_types): _tt_tmp = _tt_utf8 (_tt_tmp) # <string>:3 Else: _tt_tmp = _tt_utf8 (s TR (_tt_tmp)) # <string>:3 _tt_tmp = _tt_utf8 (Xhtml_escape (_tt_tmp)) # <string>:3 _tt_append (_tt_ TMP) # <string>:3 _tt_append (' \ n ') # <string>:4 Pass # <string>:2 _tt_append (' \ n ') # <string>:5 Pass # <string>:5 Else: # <string>:5 _tt_append (' \nno one\n ') # <string> : 7 Pass # <string>:1 _tt_append (' \ n ') # <string>:8 return _tt_utf8 ('). Join (_tt_buffer) # <strin g>:0 

Yes, you are not mistaken, Tornado compilation is to translate it into a block of code, and finally through the exec passed the parameter namespace we gave to execute the _tt_execute function.
There are 4 predefined node nodes in our template, _controlblock,_expression,_text, each node has its own way of generating.
For example, the _expression expression node, the {{name}} in our template, finds that the "{" or "{" After _parse parsing is considered an expression node,

 class _expression (_node): Def __init__ (self, Expression, line, Raw=false): Self.ex pression = Expression Self.line = line Self.raw = raw def generate (self, writer): Writer.write_line ("_ 
             tt_tmp =%s "% self.expression, Self.line) writer.write_line (" If Isinstance (_tt_tmp, _tt_string_types): " 
    "_tt_tmp = _tt_utf8 (_tt_tmp)", Self.line) Writer.write_line ("Else: _tt_tmp = _tt_utf8 (str (_tt_tmp))", Self.line) If not Self.raw and Writer.current_template.autoescape isn't None: # in Python3 functions like Xhtml_escape re 
      Turn Unicode, # So we have to convert to utf8 again. 
    Writer.write_line ("_tt_tmp = _tt_utf8 (%s (_tt_tmp))"% Writer.current_template.autoescape, Self.line) Writer.write_line ("_tt_append (_tt_tmp)", Self.line) 

The Generate method of the node is invoked at the end of the build, and Self.expression is the name above, so the value of name is append to the internal list when exec.
such as if,for are control nodes, they are defined as follows:

Class _controlblock (_node): 
  def __init__ (self, statement, line, Body=none): 
    self.statement = statement 
    Self.line = line 
    self.body = Body 
 
  def each_child (self): return 
    (Self.body,) 
 
  def generate (self, writer) : 
    writer.write_line ("%s:"% self.statement, self.line) with 
    writer.indent (): 
      self.body.generate ( Writer) # Just in case the body is 
      empty 
      

The Generate method of the control node is a bit meaningful because if,for is the next line that needs to be indented, so call the with Writer.indent continue indent control, you can look at
The indent method of _codewriter.
The interesting thing in the node is _extendsblock, which is the node that implements the goal,

Class _extendsblock (_node): 
  def __init__ (self, name): 
    self.name = name 

We found that the Generate method was not defined, and would it not be an error when generating an inherited node? Let's take a look at some examples

Loader = Loader ('. ') 
T=template ("" \ 
{% extends base.html%} 
{% block login_name%}hello world! {{Name}} {% END%} "" 
, Loader=loader) 

The current directory under Base.html is as follows:

We can look at the parsed node,

Since we inherited the base.html, we should build it as a base.html template and replace the block in base.html with the newly defined block.
This is a very normal way of thinking, Tornado is really doing so, but not to deal with the _extendsblock.
And in the _generate_python of real template

 def _generate_python (self, loader, compress_whitespace): buffer = Stringio () Try: # 
     Named_blocks maps from names to _namedblock objects named_blocks = {} ancestors = Self._get_ancestors (loader) 
     Ancestors.reverse () for ancestor in Ancestors:ancestor.find_named_blocks (loader, named_blocks) writer = _codewriter (buffer, named_blocks, loader, ancestors[0].template, compress_whitespace) ances Tors[0].generate (writer) return Buffer.getvalue () finally:buffer.close () def _get_ancestors (self, LOA 
       Der): Ancestors = [Self.file] for chunk in self.file.body.chunks:if isinstance (Chunk, _extendsblock): 
       If not loader:raise parseerror (' {% extends%} block found, but no ' "template loader ') Template = Loader.load (Chunk.name, Self.name) ancestors.extend (template._get_ancestors (loader)) return Cestors 

_generate_python call _get_ancestors to get the parent template for the current template, we see if there are _ in the _file node of the current template Extendsblock represents a parent template and loads the parent template through Loader.load, at which point the parent template is already a resolved _file node. So, in the above template, ancestors is [the current template _file node, the parent template _file node],ancestors.reverse () actually ancestors[0] is the parent template, we see the last is through Ancestors[0]. Generate (writer) to generate the code. How does the current template replace the block content of the parent template?
Look at the graph, block login_name by parsing to _namedblock, in _generate_python by calling Ancestor.find_named_blocks to replace the
The _namedblock of the parent template.

For ancestor in ancestors:
    ancestor.find_named_blocks (loader, named_blocks)
ancestor is actually _file node, find_named _blocks will traverse all nodes in the _file node and invoke Find_named_blocks
 
class _namedblock (_node): 
  def find_named_blocks (self, loader, Named_blocks): 
    named_blocks[self.name] = self 
    _node.find_named_blocks (self, loader, named_blocks) 

The other node find_named_blocks did nothing, _namedblock named_blocks[self.name] = self replaced with the _namedblock of the current template because the Ancestors parent template was in front The current template is behind, so the last use is the _namedblock of the current template.
After the code is generated generate will exec code in the given namespace

def generate (self, **kwargs): "" "Generate this template with the given arguments." " Namespace = {"Escape": Escape.xhtml_escape, "Xhtml_escape": Escape.xhtml_escape, "Url_escape": Escape.url_ Escape, "Json_encode": Escape.json_encode, "Squeeze": Escape.squeeze, "linkify": escape.linkify, "date 
    Time ': datetime, ' _tt_utf8 ': Escape.utf8, # for internal use ' _tt_string_types ': (Unicode_type, Bytes_type), 
    # __name__ and __loader__ allow the traceback mechanism to find # The generated source code. "__name__": Self.name.replace ('. ', ' _ '), "__loader__": Objectdict (Get_source=lambda name:self.code), Namespa Ce.update (Self.namespace) namespace.update (Kwargs) exec_in (self.compiled, namespace) Execute = namespace["_tt_exe Cute "] # Clear the Traceback module ' s cache of source data now, we ' ve generated a new template (mainly for thi S module ' s # unittests, where different tests reuse the same name.
  Linecache.clearcache () return execute () 
 

So you can use DateTime in a template, all by injecting it into the template and, of course, by
Get_template_namespace injected into the web.py.

 def get_template_namespace (self): "" " 
   Returns a dictionary to be used as the default Template namespace. 
 
   May is overridden by subclasses to add or modify values. 
 
   The results is combined with additional defaults in the 
   ' tornado.template ' module and keyword Argu ments to 
   ' render ' or ' render_string '. 
   " 
   " " namespace = Dict ( 
     handler=self, 
     request=self.request, 
     current_user=self.current_user, 
     locale= Self.locale, 
     _=self.locale.translate, 
     static_url=self.static_url, 
     xsrf_form_html=self.xsrf_form _html, 
     reverse_url=self.reverse_url 
   ) 
   namespace.update (self.ui) return 
   namespace 

Let's take a look at how the tornado template supports the UI module.

{% for entry in entries%} 
 {% Module Entry (Entry)%} 
{% END%} 

_module node will be generated when using module

Class _module (_expression): 
  def __init__ (self, Expression, line): 
    super (_module, self). __init__ ("_tt_ Modules. "+ expression, line, 
                   raw=true) 

We see that the _module node is actually inherited from the _expression node, so the final execution is _tt_modules. Entry (Entry)
_tt_modules is defined in the RequestHandler of web.py.

self.ui["_tt_modules"] = _uimodulenamespace (self,application.ui_modules)

and injected into the template through the get_template_namespace above.

Class _uimodulenamespace (object): "" " 
  Lazy namespace which creates uimodule proxies to a bound. 
  " " def __init__ (self, Handler, ui_modules): 
    Self.handler = handler Self.ui_modules 
    = Ui_modules 
 
  def __getitem __ (self, key): Return 
    self.handler._ui_module (key, Self.ui_modules[key]) 
 
  def __getattr__ (self, key): 
    Try: Return 
      Self[key] 
    except Keyerror as e: 
      raise Attributeerror (str (e)) 

So when performing _tt_modules. Entry (Entry) first accesses the __getattr__ of _uimodulenamespace, then accesses __getitem__, and finally calls
Handler._ui_module (Key, Self.ui_modules[key]),

def _ui_module (self, Name, module): 
  def render (*args, **kwargs): 
    if not hasattr (self, "_active_modules"): 
      Self._active_modules = {} 
    if name not in Self._active_modules: 
      self._active_modules[name] = module (self) 
    rendered = Self._active_modules[name].render (*args, **kwargs) return 
    rendered return 
  render 

_tt_modules. Entry (Entry) Entry will be passed to _ui_module's inner render, which is args=entry
Self._active_modules[name] = module (self) is now the instantiated uimodule, invoking render to get the rendered content

Class Entry (Tornado.web.UIModule): 
  def render (self, Entry, Show_comments=false): Return 
    Self.render_ String 
      

Of course, if you find it troublesome to do so, you can also use Tornado's own Templatemodule, which inherits from Uimodule,
You can use that.

{% Module Template ("module-entry.html", Show_comments=true)%} 

In module_entry.html, you can refer to the required static file through Set_resources

{{set_resources (embedded_css=. Entry {margin-bottom:1em;}}}} 

Note here that you can only use the Set_resources function in the HTML file referenced by template, because set_resources is the internal function of Templatemodule.render

Class Templatemodule (Uimodule): "" "uimodule that simply renders given the template.  {% Module Template (' foo.html ')%} is similar to {% include ' foo.html '%}, but the module version gets its own namespace 
 
  (with Kwargs passed to Template ()) instead of inheriting the outer Template ' s namespace. Templates rendered through this module also get access to Uimodule ' s automatic javascript/css. Simply call set_resources inside the template and give it keyword arguments to the corresponding on methods : {{set_resources (Js_files=static_url ("My.js")}} Note that this resources are output once each template file, not on 
  Ce per instantiation of the template, so they must not depend on any arguments to the template. "" "Def __init__ (self, Handler): Super (Templatemodule, self). __init__ (handler) # Keep resources in both a LIS T and a dict to preserve order self._resource_list = [] Self._resource_dict = {} def render (SELF, PATH, **kwargs): Def set_resources (**kwargs): If Path not in Self._resource_dict:self._resource_ List.append (Kwargs) Self._resource_dict[path] = Kwargs else:if Self._resource_dict[path]!= Kwar Gs:raise valueerror ("set_resources called with different" "Resources for" same templat  E ") return" "Return self.render_string (Path, set_resources=set_resources, **kwargs)