A brief introduction to Sybase massive data storage, access and management

--sybase vlds (Very Large Data Store) solutions and success stories

Mass data is a reality in business today

With the improvement of information level, the data has gone beyond its original category, it contains various kinds of data information such as business Operation data, report statistic data, Office document, email, hypertext, form, report and picture, audio and video etc. People use massive amounts of data to describe huge, unprecedented, and growing data.

Mass data is a reality facing business today. Any enterprise in the face of its enterprise database due to the heavy burden of expansion, improve the capacity of massive data access and business analysis of the requirements also become more and more urgent.

· Data explosion. Today, the data that needs to be managed by the enterprise is growing at a exponentially faster rate. Analysts found that companies collect, store, and analyze data on customers, finance, products, and operations at a rate of 125% per cent. All the factors that led to the explosion of the data, for example: Network application increases the growth rate of data, monitor click Flow need to store more and more different data types than ever before; Multimedia data also increases storage requirements; We store and manage not just numbers and text, but also video, audio, images, Temporary data and more, the growth of these data is increasing; Data warehousing and data mining applications encourage businesses to store more and more data over time and for longer periods. The result of these actual situations is that the data increase substantially.

· Regulatory requirements. In the wake of the accounting scandal, lawmakers and policymakers have imposed stringent new demands that affect almost every big business around the world. The bill requires public companies to comply with strict financial record keeping and reporting regulations. If companies fail to access accurate financial information in a timely and reliable manner, they face a threat of fines, investigations, prosecutions and even tougher declines in shareholder confidence. This requires companies to provide uninterrupted access to more data and more analysis, which is bound to prolong the maintenance cycle of data and increase data capacity.

· The need for unstructured data applications. Structured data refers to data such as enterprise financial accounts, customer information, business operation data and so on with obvious structural characteristics. Unstructured data includes scanning document images, faxes, photos, computer-generated reports, word processing documents, spreadsheets, presentations, voice and video clips, and more. According to the Industry Analysis report, unstructured data occupies more than 85% of the total amount of information, the data volume and its huge, is the core of the resource management. In the same way, enterprises need to store, retrieve, filter, extract, excavate and analyze unstructured data.

· The need for historical data archiving and access. Generally speaking, business systems are used to deal with business transactions, to keep the performance of these critical business systems from being severely impacted, business systems often hold only business transaction data for a short period of time, a large amount of historical data is backed up to tape, or moved to a static save on another storage device when SQL is run on them Queries to recover them from the archive environment. However, with the increasing importance of the data and the application of data analysis and data mining, the access of historical data will become important, frequent and direct. The continuous accumulation of historical data also puts forward new demands for the storage, management and access of massive data.

· The need for data integration and data analysis. At present, the storage of enterprise information has the characteristics of data structure diversification and storage isomerization, the data of enterprise can be stored in traditional system, large data warehouse or data Operation Island with billing, ordering, manufacturing, distribution or other functions, so it will bring great difficulty to visit. Data integration and data analysis have become the hotspot of the application of information management technology. Only on the basis of effective data integration, can we eliminate the information islands, reduce the difficulty of obtaining effective information, and obtain the necessary information based on the analysis and processing of the integrated data.

Traditional relational databases face greater challenges

The traditional relational database is an important milestone in the development of computer data management, which has the advantages of data structure, minimum redundancy, high program and data independence, easy to expand, easy to compile application and so on, and the larger information system is based on the structural database design.

However, as more and more enterprises produce large amount of data, especially the development of Internet and intranet technology, the application of unstructured data is increasing, and the demand of fast access to massive data, effective backup and recovery mechanism, real-time data analysis, etc. The traditional relational database has been developed since 1970, but there are still many deficiencies in coping with the mass data processing although the function is perfect.

Lack of fast access to massive amounts of data

When your competitor announced a new pricing system in the afternoon of Friday, the president of your organization wants an analysis of the impact of your company before the morning of Monday, and the last thing the business analyst wants to do is spend 20 minutes waiting for the entire table scan and multiple table connections to get "if ..." What will happen to "the query. Because there are no optimized queries that can take a long time, the needs of the queried users need to be scheduled, multiple queries compete for CPU resources, and business requirements are often changed. All of these require constant tuning of the database or even redesigning the database.

Lack of massive data access flexibility

In reality, users want to have great flexibility when querying. Users can ask any questions, can ask questions about any data, and can ask questions at any time. Whatever questions are raised, they can be answered quickly. Traditional databases do not provide a flexible solution, and cannot respond quickly to random queries, as it waits for system administrators to tune special queries, which causes many companies not to have this rapid response capability.

Weak data-processing capacity for unstructured

The traditional relational database processing of data types is limited to numbers, characters and so on, and the processing of multimedia information only stays in the storage of simple binary code files. However, with the improvement of users ' application demand, the development of hardware technology and the colorful multimedia communication method provided by Intranet/internet, the requirement of multimedia processing is increased from simple storage to recognition, retrieval and deep processing, so how to deal with the sound, image, and 85% of the total information Time series signal and video, e-mail and other complex data types, is a lot of database manufacturers are facing problems.

Massive data leads to increased storage costs, maintenance and management costs

Large enterprises are facing the pressure of business and it input, compared with the past, the performance/price ratio of the system is more concerned. The GIGA study shows that ROI is becoming more and more important. Massive amounts of data allow businesses to invest heavily in storage hardware because of large amounts of online data and data bloat, and while the cost of storage devices is falling, the overall cost of storage is increasing and is becoming one of the largest it expenditures. On the other hand, massive data has led DBAs into ongoing database management and maintenance efforts.