Data Warehouse

Based on the hive of the Water conservancy Census Data Warehouse Chen Wandingshen Gu Xinzhen According to the mass and multidimensional characteristics of water Conservancy census data, this paper studies the Hadoop and hive developed rapidly under the concept of "big data", and combines the mature technology of the traditional data warehouse in multidimensional data analysis, This paper puts forward the construction method of the data warehouse of the Water conservancy census based on hive, describes the architecture of the Data Warehouse system, and according to the design characteristics of the hive, improves the traditional multidimensional analysis model through the method of splitting bucket, reducing dimension table and redundant fact table, finally building the cluster system to ...

Most data warehousing projects will bring huge investment. For example, the Nordbanken bank, a European lender, spent 100 million euros to build a data warehouse system for data unification across regions and business units. However, in the face of the same business challenges, another European bank, UniCredit, has taken a radically different approach. "Our philosophy is to look for space in other projects, as long as there are any other projects," UniCredit chief financial officer Criscito Ambrisi said in an interview.

Bill Inmon and Ralph Kimball, who were exposed to two names at school, were unfamiliar to most of the two Americans, but they were a resounding figure in the database field. Bill Inmon, known as the "Father of the Data Warehouse", he can now see a lot of scholarly papers and articles on the Web, and Wikipedia's introduction to him should be very comprehensive: in the 80 's, Inmon's "Data Warehouse" book defines the concept of data warehousing, Then gave more ...

Discussion on the construction of large data warehouse platform with cloud computing technology horse and good analysis of the current telecom operators in the construction of large data warehouses in the infrastructure faced by the technical problems, combined with the technical characteristics of cloud computing, this paper gives a solution of cloud computing technology and compares it with the traditional scheme, and analyzes the advantage factors of cloud computing scheme. Keywords: cloud computing technology, data Warehouse, large-scale parallel processing, column storage temp_12090709573798.p ...

Data quality (Quality) is the basis of validity and accuracy of data analysis conclusion and the most important prerequisite and guarantee. 　　Data quality assurance (Quality Assurance) is an important part of data Warehouse architecture and an important component of ETL. We usually filter dirty data through data cleansing to ensure the validity and accuracy of the underlying data, and data cleaning is usually the front link of data entry into the Data warehouse, so the data must be ...

The construction of Clinical medical Information Analysis system based on cloud computing Du Shouhong Wang Guozhong Joey Research on the collection, analysis, integration and storage of clinical medicine information of public cloud, which is based on the common network and medical professional website, constructs the medical data Warehouse and data multidimensional cube based on standardization and standardization, and on this basis, The research platform of medical data mining is constructed by using data stored in data Warehouse to carry out multi-level and multi-angle mining work and integration analysis, and to provide the foundation of attention to medical and health personnel in the discipline with Software as service (SaaS) model.

Research on the framework of large data analysis process Jinzongzefonyali Yang Zhengnan Zhang with the continuous innovation of information technology, the constant expansion of the volume of data has become a topic closely related to daily life. The value of digging up large data is already hot, and how to analyze large data more efficiently and quickly has become one of the major challenges of big data development. In recent years, academia and industry have studied the analysis of large data and obtained some research results, but the research on large data analysis is still very limited. In this paper, the traditional data warehouse and data warehouse of large data era are firstly ...

In the past, Hadoop seemed to be synonymous with big data. But with the recent deepening of large data applications, it has become increasingly popular to just think of it as a storage tool for large data. But that's not necessarily a bad thing. Taking Hadoop as a cheap and efficient storage is just the perfect starting point for the next phase of Hadoop's evolution. The Hadoop 2.0, which is to be unveiled this summer, will make the information in the Data warehouse and the unstructured data pool more accessible than ever before. Hadoop bucket Since becoming a big data tool, Hadoop is a ...

The example demonstrates using the IBM BCU design architecture to benchmark TPC as data source (300GB data volume) and test case, showing the pull effect of "troika" on query performance. Whether in the POC test or in the real production system, query performance is an important indicator of customer concern. Through this article, the reader can fully understand the "troika" of the mystery, the text of the example demo to the reader has reference and referential significance. In the Http://www.aliyun.com/zixun/aggreg ...

Talend open Studio for data integration is an open source data extraction, conversion and loading tool (ETL). It can perform synchronization from http://www.aliyun.com/zixun/aggregation/8302.html "> Data Warehouse to Database, as well as file format conversion, with a graphical interface based on Eclipse RCP, Generate a Perl or Java related number script. The 8639 ....

Design and realization of network identity recognition system based on hadoop/hive architecture Nanjing University of Posts and telecommunications Shingwen based on the actual system development, this paper summarizes the design and implementation of a network identity recognition system based on hadoop/hive architecture. After the raw data of each data source is cleaned layer by mapreduce, it is loaded into a new event-based data warehouse. Then, using the HIVEQL language, under the control of the Professional Workflow Control tool, the data analysis and processing work are completed according to the user's requirements. At last...

Research and implementation of distributed ETL based on Hadoop Platform Donghua University gang the author mainly studies and realizes the work as follows first, distributed ETL Framework design. Based on the theory of dimension modeling in Data Warehouse, a distributed ETL framework including dimension and fact parallel processing and HDFS data block allocation is designed by analyzing the MapReduce working mechanism and job scheduling under the Hadoop platform. Second, the study of the parallel processing of facts. From the fact table lookup Surrogate key and the multi-granularity fact pre-aggregation two angles to start, put forward in the gradient ...

The machine learning algorithm platform allows users to experiment by dragging visualized operational components so that engineers without a machine learning background can easily get started with data mining.

Hadoop has long been a synonym for big data. However, with the recent development of the application of big data, everyone has become more and more inclined to regard it as a storage tool for big data. But this is not necessarily a bad thing. Hadoop as cheap and effective storage is just the perfect starting point for the next phase of Hadoop's evolution. Hadoop 2.0, to be unveiled this summer, will make information in the data warehouse and unstructured data pools as easy to access as ever. Hadoop vats have become big data tools since ...

This article is for IBM system architect Mr. Wang Wenjie (valen_won@hotmail.com) in its blog park Zhiding Bowen thinking EXADATA (link original address: http://www.cnblogs.com/wenjiewang/archive/2012/10/07/ 2714406.html) mentioned some of the views on the Exadata from the technical point of view gives my personal opinion of some different, of course, I have limited level, it is inevitable that omissions or mistakes. ...

Large data analysis and high speed data update Chen Shimin large data for data management system platform The main challenges can be summed up as volume (large data), velocity (data generation, acquisition and update speed) and produced (a wide range of data) 3 aspects. For a large data analysis system, try to understand the importance of velocity and how to deal with the challenge of velocity. First, compare the different requirements of the data processing, data flow, and analysis system to velocity. Then from the data update and the large data analysis system correlation ...

Recently released Hive 0.13 ACID semantic transaction mechanism used to ensure transactional atomicity, consistency and durability at the partition layer, and by opening Zoohttp: //www.aliyun.com/zixun/aggregation/19458.html "> Keeper or in-memory lock mechanism to ensure transaction isolation.Data flow intake, slow changes in dimension, data restatement of these new use cases in the new version has become possible, of course, there are still some deficiencies in the new Hive, Hive ...

Hadoop technology friends will certainly be confused about its system under the parasitic open-source projects confused, and I promise Hive, Pig, http://www.aliyun.com/zixun/aggregation/13713.html "> HBase these open source Technology will get you some confused, do not confused more than just one, such as a rookie post doubt, when to use Hbase and when to use Hive? ...

There is no doubt that the big data age has come. So how do we deal with this situation? 　　Now, let's hear what the experts with the experience say. First, we need to know how to make the most of the big data in hundreds of terabytes of information. It all depends on the individual's needs and preferences. Interclick Advertising Services has found a way to provide more efficient solutions while providing near-real-time data analysis. Harvard Medical School also learned that in terms of the number of patients and years to keep the ...

Hadoop technology friends will certainly be confused about its system under the parasitic open-source projects confused, and I promise Hive, Pig, http://www.aliyun.com/zixun/aggregation/13713.html "> HBase these open source Technology will get you some confused, do not confused more than just one, such as a rookie post doubt, when to use Hbase and when to use Hive? ...