A detailed explanation of how Python probes are implemented

This article will briefly explain how the Python probe is implemented. and in order to verify this principle, we will also implement a simple probe program that specifies the time of the function execution.

The implementation of the probe mainly involves the following knowledge points:

Sys.meta_path
sitecustomize.py
Sys.meta_path

Sys.meta_path This simple is the ability to implement the import hook,
When an import-related operation is performed, the object defined in the Sys.meta_path list is triggered.
For more detailed information about Sys.meta_path, refer to the Sys.meta_path content in the Python documentation and
PEP 0302.

Objects in the Sys.meta_path need to implement a Find_module method,
This find_module method returns None or an object that implements the Load_module method
(The code can download part1 from GitHub):

Import SYS class Metapathfinder:   def find_module (self, FullName, Path=none):    print (' Find_module {} '. Format ( FullName))    return Metapathloader () class Metapathloader:   def load_module (self, fullname):    print (' Load_ module {} '. Format (fullname))    sys.modules[fullname] = sys    return sys sys.meta_path.insert (0, Metapathfinder () if __name__ = = ' __main__ ':  import http  print (HTTP)  print (Http.version_info)

The Load_module method returns a Module object, which is the Import module object.
For example, I would like to replace HTTP with the SYS module.

$ python meta_path1.py
Find_module http
Load_module http

Sys.version_info (major=3, minor=5, micro=1, releaselevel= ' final ', serial=0)
With Sys.meta_path we can implement the functions of the import hook:
When import a predetermined module, the object in this module to a civet cat for the Prince,
This enables the acquisition of probe information such as the execution time of a function or method.

Above said the cat for the Prince, then how to a cat for a cat to change the operation of the prince?
For function objects, we can use adorners to replace function objects (code can download part2 from GitHub):

Import functoolsimport Time def func_wrapper (func):  @functools. Wraps (func)  def wrapper (*args, **kwargs):    print (' Start func ')    start = Time.time ()    result = Func (*args, **kwargs)    end = Time.time ()    print (' spent {}s '. Format (end-start))    return result  return wrapper def sleep (n):  time.sleep (n)  return n if __ name__ = = ' __main__ ':  func = func_wrapper (sleep)  print (func (3))

Execution Result:

$ python func_wrapper.pystart funcspent 3.004966974258423s3

Let's implement a function that calculates the execution time of a specified function of a specified module (the code can download part3 from GitHub).

Suppose our module file is hello.py:

Import Time Def sleep (n):  time.sleep (n)  return n

Our import hook is hook.py:

Import Functoolsimport importlibimport sysimport Time _hook_modules = {' Hello '} class metapathfinder:def Find_module (SE LF, FullName, Path=none): print (' Find_module {} '. Format (fullname)) if FullName in _hook_modules:return Metapat Hloader () class Metapathloader:def load_module (Self, fullname): print (' Load_module {} '. Format (fullname)) # ' sys. Modules "is stored in the imported module if FullName in Sys.modules:return Sys.modules[fullname] # First remove the custom from the Sys.meta_path The Semantic Finder # prevents the following execution of the import_module when triggering this finder # so that there is a problem with recursive calls finder = Sys.meta_path.pop (0) # import module mo Dule = Importlib.import_module (fullname) Module_hook (FullName, module) sys.meta_path.insert (0, finder) return m Odule sys.meta_path.insert (0, Metapathfinder ()) def module_hook (FullName, module): if FullName = = ' Hello ': Module.slee p = Func_wrapper (module.sleep) def func_wrapper (func): @functools. Wraps (func) def wrapper (*args, **kwargs): Print (' st Art func ') start = TiMe.time () result = Func (*args, **kwargs) end = Time.time () print (' spent {}s '. Format (End-start)) return Resul T return wrapper

Test code:

>>> Import hook>>> Import hellofind_module helloload_module hello>>>>>> Hello.sleep (3) Start funcspent 3.0029919147491455s3>>>

In fact, the above code has implemented the basic function of the probe. But one problem is that the code above needs to be displayed.
The import hook operation does not register the hooks we define.

So is there a way to automate the import hook operation when the Python interpreter is started?
The answer is that this can be done by defining sitecustomize.py.

sitecustomize.py
Simply put, the Python interpreter initializes the sitecustomize and Usercustomize modules that are present in the import PYTHONPATH:

The directory structure of the experimental project is as follows (code can download part4 from GitHub)

$ tree
.
├──sitecustomize.py
└──usercustomize.py
sitecustomize.py:

$ cat sitecustomize.py
Print (' This is Sitecustomize ')
usercustomize.py:

$ cat usercustomize.py
Print (' This is Usercustomize ')
Add the current directory to PYTHONPATH and see the effect:

$ export pythonpath=.$ pythonthis is sitecustomize    <----This is usercustomize    <----Python 3.5.1 (default, DEC, 17:20:27) [GCC 4.2.1 Compatible Apple LLVM 7.0.2 (clang-700.1.81)] on Darwintype "help", "copyright", "credits "or" license "for more information.>>>

You can see that the import is indeed automatic. So we can change the previous detection program to support automatic execution of the import hook (the code can download part5 from GitHub).

Directory structure:

$ tree
.
├──hello.py
├──hook.py
├──sitecustomize.py
sitecustomize.py:

$ cat Sitecustomize.pyimport Hook

Results:

$ export pythonpath=.$ pythonfind_module Usercustomizepython 3.5.1 (default, Dec, 17:20:27) [GCC 4.2.1 Compatible A Pple LLVM 7.0.2 (clang-700.1.81)] on Darwintype ' help ', ' copyright ', ' credits ' or ' license ' for more Information.find_modu Le readlinefind_module atexitfind_module rlcompleter>>>>>> import hellofind_module Helloload_ Module hello>>>>>> hello.sleep (3) Start funcspent 3.005002021789551s3

However, the above detection program actually has a problem, that is, the need to manually modify the PYTHONPATH. A friend with a probe program should remember that using a probe like Newrelic is just a command to execute: newrelic-admin run-program python hello.py actually modifies Pythonpath's operation in Newreli C-admin is done in this program.

Here we also want to implement a similar command-line program, called Agent.py Bar.

Agent
or modify it on the basis of the previous program. First, adjust a directory structure, put the hook operation in a separate directory, easy to set up Pythonpath after the other interference (the code can be downloaded from GitHub part6).

$ mkdir bootstrap$ mv hook.py bootstrap/_hook.py$ Touch bootstrap/__init__.py$ touch agent.py$ tree.├──bootstrap│ ├─  ─__init__.py│  ├──_hook.py│  └──sitecustomize.py├──hello.py├──test.py├──agent.py

The content of the bootstrap/sitecustomize.py is modified to:

$ cat bootstrap/sitecustomize.py
Import _hook
The contents of agent.py are as follows:

<span class= "kn" >import</span> <span class= "nn" >os</span><span class= "kn" >import< /span> <span class= "nn" >sys</span> <span class= "n" >current_dir</span> <span class= "O" >=</span> <span class= "n" >os</span><span class= "O" >.</span><span class= "n" > Path</span><span class= "O" >.</span><span class= "n" >dirname</span><span class= "P" > (</span><span class= "n" >os</span><span class= "O" >.</span><span class= "n" > Path</span><span class= "O" >.</span><span class= "n" >realpath</span><span class= "p "> (</span><span class=" n ">__file__</span><span class=" P ">)) </span><span class = "n" >boot_dir</span> <span class= "o" >=</span> <span class= "n" >os</span><span Class= "O" >.</span><span class= "n" >path</span><span class= "O" >.</span><span class= "n" >join</span><span class= "P" > (</span><span class= "n" > Current_dir</span><span class= "P" >,</span> <span class= "s" > ' Bootstrap ' </span>< Span class= "P" >) </span> <span class= "K" >def</span> <span class= "NF" >main</span> <span class= "P" > ():</span> <span class= "n" >args</span> <span class= "O" >=</span> <span class= "n" >sys</span><span class= "O" >.</span><span class= "n" >argv</span>  <span class= "P" >[</span><span class= "Mi" >1</span><span class= "P" >:]</span> <span class= "n" >os</span><span class= "O" >.</span><span class= "n" >environ</span ><span class= "P" >[</span><span class= "S" > ' PYTHONPATH ' </span><span class= "P";] </span> <span class= "o" >=</span> <span class= "n" >boot_dir</span> <span clAss= "C" ># executes the following Python program command </span> <span class= "C" ># sys.executable is the absolute path of the Python interpreter program ' which Python ' &l t;/span> <span class= "C" ># >>> sys.executable</span> <span class= "C" ># '/usr/local/var/ pyenv/versions/3.5.1/bin/python3.5 ' </span> <span class= "n" >os</span><span class= "O" >.< /span><span class= "n" >execl</span><span class= "P" > (</span><span class= "n" >sys </span><span class= "O" >.</span><span class= "n" >executable</span><span class= "P" >,</span> <span class= "n" >sys</span><span class= "O" >.</span><span class= "n" >executable</span><span class= "P" >,</span> <span class= "O" >*</span><span class = "n" >args</span><span class= "P" >) </span> <span class= "K" >if</span> <span class= "N" >__name__</span> <span class= "o" >==</span> <span class= "S "> ' __main__ ' </span><span class=" P ">:</span> <span class=" n ">main</span><span class= "P" > () </span>

The contents of the test.py are:

$ cat Test.pyimport sysimport Hello print (sys.argv) print (Hello.sleep (3))

How to use:

$ python agent.py test.py arg1 arg2find_module usercustomizefind_module helloload_module hello[' test.py ', ' arg1 ', ' arg2 ' ]start funcspent 3.005035161972046S3

At this point, we have implemented a simple Python probe program. Of course, compared with the actual use of the probe program is definitely a big gap, this article is mainly to explain the probe behind the implementation of the principle.

If you are interested in the specific implementation of the commercial probe program, you can look at the foreign New Relic or domestic OneAPM, Tingyun and so on these APM manufacturers of commercial Python probe source code, I believe you will find some very interesting things.