Implementation of Data Quality Monitoring

In the process of data generation, we usually judge whether the data is generated based on whether the script is executed successfully or not. However, there is a problem in this logic: we can only judge whether there is data, but not monitor the data quality. Let's take a simple example: we have a script to generate daily traffic data. On the day of 20190410, 100W data was generated, and on the day of 20190411, only 50W data was generated. This is obviously abnormal. In this case, the problem must be checked immediately, otherwise all reports relying on this data will have problems. Therefore, data quality monitoring is necessary. How to do this? Take this example to try to solve the problem. 

We define some concerned indicators, such as the variables daybeforeyesterday, yesterday, each of which is followed by a calculated value SQL; with these variables, the next step is to write expressions

Math.abs (${yesterday}-${dayBeforeyesterday})/${dayBeforeyesterday}>0.1

This expression uses JavaScript syntax. The meaning of this expression: if the error between yesterday's data amount and the day before yesterday's data amount exceeds 10%, an alarm will be given; how to parse and execute the judgment expression in the background? In fact, it can be used in Java javax.script.ScriptEngine To achieve this function, that's why expressions need to use JavaScript syntax. Look at the code

ScriptEngineManager scriptEngineManager = new ScriptEngineManager();
ScriptEngine scriptEngine =  scriptEngineManager.getEngineByName ("javascript");
String expression = " Math.abs (${yesterday}-${dayBeforeyesterday})/${dayBeforeyesterday}>0.1";
Map<String,String> values = new HashMap<String,String>();
values.put ("dayBeforeyesterday","1000340");
values.put ("yesterday","1110340");
Set<String> strings =  values.keySet ();
for(String k:strings){
expression =  expression.replaceAll ("\\$\\{"+k+"}", values.get (k));
}
System.out.println ("expression:" + expression);
try {
String result =  String.valueOf ( scriptEngine.eval (expression));
System.out.println (result);
} catch (ScriptException e) {
e.printStackTrace();
}

output

expression: Math.abs (1110340-1000340)/1000340>0.1
true