Data Integration: What is Data Integrity

Data integrity is the guarantee of data accuracy and consistency during the entire process of the data life cycle (from recording data to destroying data). In simple terms, data integrity means that you have recorded the data as expected, and that it has not accidentally appeared throughout the life cycle. The concept is simple, but practice is not. Data integrity is one of the important joints of any big data system.

Importance of data integrity
Data integrity is very important because almost every key business decision is based on company data. With good data integrity, you can analyze the company's data to answer the following questions: How is your business growing? What is the cost of your business? How are your sales in different regions? What in your business is not making ends meet? What is the efficiency of the different departments of the company? Has your business reached the benchmark target? Obviously, it is difficult for us to answer these comprehensive questions without complete data.

Challenges to data integrity
The challenges to data integrity fall into two broad categories:

Physical disaster recovery
The physical system storing the data may suffer any physical damage: fire, flood, explosion, extreme temperature, radiation, corrosion, and any natural disasters that may affect the physical structure may affect the integrity of your physical data.

Human error
Software is written and used by humans. Humans are more prone to errors than machines. Therefore, when humans intervene in the data system, the logical integrity of the data may be destroyed. In addition to improper human operation, there are several possibilities:

The design of the software failed to include appropriate constraints

Software errors allow incorrect data to be introduced or deleted from the system

Transmission errors may occur during data transmission, including accidental changes or data leakage during transmission from one device to another

Data modification may be deliberately malicious, such as hacking, or viruses and malware





How to maintain data integrity
Physical data integrity
Deploy a horizontally scalable system and set up at least 3 partition backups so that the backup system can be used when a system fails. We recommend that it is best handled by cloud vendors. If you need to maintain your own machine, you need to ensure the power supply, configure anti-radiation chips, and routinely check the fire and anti-theft systems.

Non-physical data integrity
Maintaining data integrity may involve multiple systems and steps. You need to plan an integrity inspection program. The area to be inspected includes: input system, storage system and all data consumers. Since most company data is stored in a data warehouse or database, the technical team needs to regularly check the data permissions of all members and departments and the binding force between data and data.