JINJA2 batch generation of Python scripts

? In the process of using airflow, a large number of DAG scripts are required for performance testing, and if you write a DAG script a bit too much trouble, then think of the Python jinja2 template engine to implement batch script generation.

Install the JINJA2 module with the PIP command first:

$ pip install jinja2

Then create the template file (the template can be any form of text format without a specific extension or even a name extension):

dag_template

From datetime import Timedelta, datetimeimport pytzfrom airflow.operators.bash_operator import Bashoperatorfrom    Airflow.operators.dummy_operator Import dummyoperatorfrom airflow.models Import Dagdefault_args = {' owner ': ' Cord ', # ' Depends_on_past ': False, ' depends_on_past ': True, # ' start_date ': Airflow.utils.dates.days_ago (2), ' Wait_for_ Downstream ': True, ' execution_timeout ': Timedelta (minutes=3), ' email ': [' [email protected] '], ' email_on_fail Ure ': false, ' email_on_retry ': false, ' retries ': 1, ' Retry_delay ': Timedelta (minutes=5),}tz = Pytz.timezone (' Asia    /shanghai ') dt = datetime (2018, 7, tzinfo=tz) Utc_dt = Dt.astimezone (PYTZ.UTC). Replace (tzinfo=none) dag = Dag ( ' {{dag_name} ', Default_args=default_args, description= ' My dag ', schedule_interval= ' */1 * * * * ', Start_dat E=UTC_DT) root = Dummyoperator (task_id= ' root ', dag=dag) for I in range: i = str (i) task = Bashoperator (tas K_id= ' Task ' +i, Bash_command= ' echo ' date ', Dag=dag) task.set_downstream (Root) 

There are two separators in JINJA2: {% ... %} and {{ ... }} , which are {% ... %} used to execute a for loop or assignment statement, which is {{ ... }} responsible for populating the values of the expression into the template. {{ ... }}the dag_id used to populate the Dag file is used here.

The template allows you to batch generate DAG script files, generating the following code:

Tool.py

import osfrom jinja2 import Environment, FileSystemLoader#获取模板env = Environment(loader = FileSystemLoader(searchpath=""))template = env.get_template("dag_template")#删除已有的生成文件for f in os.listdir("./output"):    path_file = os.path.join("./output", f)    if os.path.isfile(path_file):        os.remove(path_file)#生成新的文件for i in range(1, 101):    output = template.render({‘dag_name‘ : "benchmark%d" % i})    with open("./output/bm%d.py" % i, ‘w‘) as out:        out.write(output)

Tool.pyThe Dag script file can be generated in batches by execution.

JINJA2 batch generation of Python scripts