SpringMVC underlying data transmission verification solution, springmvc underlying Verification

SpringMVC underlying data transmission verification solution, springmvc underlying Verification

The team's project has been running normally for a long time, but sometimes there will be bugs in the near future. Currently, we have observed two scenarios: one is to submit business requests in large batches, and the other is to generate batch export files. After an error occurs, the task is executed again.

After tracking logs, it is found that some characters are lost during transmission of large data volumes in json format between servers. As a result, the receiver cannot parse the string into json correctly after obtaining the complete string. Therefore, an error is returned.

After communicating with colleagues from other teams, we found that not only do we have this problem in our projects, but we are not fighting alone.

1 problem

The http + json data transmission scheme is used between servers. During the transmission process, some json data errors cause the data recipient to report json parsing errors, and the system function fails.

A small piece of real data error is truncated below. In the transmitted json, there is a data item called departmentIdList, whose content is a long integer array.

 

The data before transmission is:

"Dimension mentidlist": [719,721,722,723,736 7, 7369,737 1, 7373,737 5, 7377]

The received data is:

"Dimension mentidlist": [719,721 '1970, 373,737]

We can see that this error causes two problems:

1. json parsing failed

2. Some valid data is lost.

After checking the system logs in detail, this is an occasional bug and only occurs when the data is transmitted is large.

2 optional solutions 2.1 ask the architecture group for help

This is the most direct solution because our project uses the environment provided by the architecture group and they need to provide a reliable underlying data transmission mechanism.

2.2 compressed transmission data

Because the data volume is large and common text is transmitted, You can compress the content before transmission. The compression ratio of common files is also high. After compression, the content length can be less than 10% of the original data, greatly reducing the probability of transmission errors.

2.3 MD5 verification of transmitted data

Transmit data as a complete data block. Before transmission, make an md5 Digest and send the original data and digest together. After receiving the data, the receiver verifies the data first, after the verification is successful, follow-up procedures are performed. If the verification fails, re-transmission or direct error reporting can be performed.

3. solution design

To completely solve this problem, an underlying solution is designed.

3.1 Design Principles

1. Applicable type: Spring MVC project. The data sender uses the RestTemplate tool class and fastjson as the json tool class.

2. Data verification. MD5 encryption can also be used with the Data Compression mechanism to reduce the amount of data transmitted.

3. Provides underlying solutions without large-scale adjustments to system code.

3.2 core design

 

Data sender, reload RestTemplate, perform md5 Digest on the data before data transmission, and transmit the original data together with the md5 Digest.

The data receiver is overloaded with javasacthttpmessageconverter. After receiving the data, MD5 verification is performed on the data.

3.3 key code of DigestRestTemplate

Abstract The original json and generate a new json object together with the original data.

Private Object digestingJson (JSONObject json) throws Exception { String requestJsonMd5 = JsonDigestUtil. createMD5 (json ); JSONObject newJson = new JSONObject (); NewJson. put ("content", json ); NewJson. put ("md5", requestJsonMd5 ); Return newJson; }

The core part of the overloaded postForEntity function. If the input parameter is JSONObject, a method is called to abstract the data and transmit it with the newly generated json.

Object newRequest = null; If (request instanceof JSONObject ){ JSONObject json = (JSONObject) request; Try { NewRequest = digestingJson (json ); } Catch (Exception e ){ } } If (newRequest = null ){ NewRequest = request; } Return super. postForEntity (url, newRequest, responseType );

 

3.4 DigestFastJsonHttpMessageConverter core code

First, the system checks whether the md5 Digest json data has been verified. Otherwise, the system directly returns the object.

Private JSONObject getDigestedJson (JSONObject json ){ If (json. size () = 2 & json. containsKey ("md5") & json. containsKey ("content ")){ String md5 = json. getString ("md5 "); String content = json. getString ("content "); Logger.info ("degested json :{}", json ); Try { String newMd5 = JsonDigestUtil. createMD5 (content ); If (newMd5.equals (md5 )){ Json = JSON. parseObject (content ); } Else { Logger. error ("md5 is not same: {}vs {}", md5, newMd5 ); Throw new RuntimeException ("content is modified "); } } Catch (Exception e ){ } } Else { Logger.info ("may not be digested json "); } Return json; }

The original data processing code adds the code to call this method.

@ Override Protected Object readInternal (Class <? Extends Object> clazz, HttpInputMessage inputMessage) Throws IOException, HttpMessageNotReadableException { JSONObject json = null; InputStream in = inputMessage. getBody (); Charset jsonCharset = fastJsonConfig. getCharset (); Feature [] jsonFeatures = fastJsonConfig. getFeatures (); Json = JSON. parseObject (in, jsonCharset, clazz, jsonFeatures ); Json = getDigestedJson (json ); Return json; }

The current Code. If data verification fails, an exception is thrown. More mechanisms can be added in the future, such as adding verification at RestTemplate. If verification fails, re-transmission is performed.

3.5 data sender project configuration

Take the Spring Boot project as an example

Define restTemplate in Main class

@ Bean (name = "restTemplate ") Public RestTemplate getRestTemplate (){ RestTemplate restTemplate = new DigestRestTemplate (); Return restTemplate; }

To call the RestTemplate code, you only need to inject the RestTemplate dependency.

@ Autowired RestTemplate restTemplate;

3.6 data recipient Project Settings

Defined in the SpringBootApplication class

@ Bean Public HttpMessageConverters fastJsonHttpMessageConverters (){ DigestFastJsonHttpMessageConverter fastConverter = New DigestFastJsonHttpMessageConverter (); FastJsonConfig fastJsonConfig = new FastJsonConfig (); FastJsonConfig. setSerializerFeatures (SerializerFeature. PrettyFormat ); FastConverter. setFastJsonConfig (fastJsonConfig ); HttpMessageConverter <?> Converter = fastConverter; Return new HttpMessageConverters (converter ); }

 

4 postscript

After testing, this solution is feasible. Code needs to be further improved to adapt to more projects and more Java technology stacks.