Python language implements Data warehouse development __python



Data Warehouse Architecture: Stg-ods-dw-rep/dm/other, Basic dimension date + product.

Use the Python language to implement the ETL work of MySQL to Oracle, file landing method.

Define HSS functions, program execution portals, define general.py public functions, and develop python.py scripts.

Data architecture, each layer based on business design specification rules

ETL work, Extract,transform,load,

Export file: The database itself exports statements,

Transform remotely get files, Python has packaged packages, and you can write your own

Load the file, the database itself statement,

Define file delimiters, define file formats

The implementation of HSS function, new HSS name sh text,

#!/usr/bin/bash

exec/usr/bin/python/$ $@ ($ is the argument script name $@ other parameter entry)

The following is the Python language design, check the environment configuration, get the parameter configuration (for SH parameters, and then pass to Python), to achieve flexible entry hss–s test.py–p etl_test–t 20170607, unlike pythontest.py arg1 arg2 Arg3 must specify the arguments for each bit.

HSS is actually an individual name, HSS = ' SH/PATH/HSS '

general.py defines a common function.

python.py script, which is a process that is called by the HSS function.

test.py

def Deal:

Try

Sql

Except

Abnormal

Template for the implementation of the Python language Data Warehouse development Annex