SQL Server Hadoop: new Big Data frontier

In the context of big data, Microsoft does not seem to be promoting their big data products or solutions as well as other database vendors. In dealing with Big Data Challenges, some Internet giants are at the forefront, such as Google and Yahoo. The former processes 20 PB of data every day, most of them are document-based index files. Of course, the definition of big data is inaccurate, and it is not limited to indexes, emails, documents, Web server logs, Social Network Information, and all other unstructured databases in an enterprise constitute part of big data.

To address the challenges of this data, the Apache Hadoop open-source platform was deployed without exception, such as Autodesk, IBM, and Facebook, and Google and Yahoo. Microsoft also noted this trend, so they added the Hadoop connector to their database platform. This connector allows enterprises to freely move massive data between Hadoop clusters and SQL Server 2008 R2, parallel data warehouses, and the latest SQL Server 2012 (Denali. Because the connector allows two-way data movement, you can not only use the powerful storage and data processing functions provided by SQL Server, but also use Hadoop to manage massive unstructured data sets.

However, traditional Microsoft users are still unfamiliar with the SQL Server Hadoop connector and are not used to it. This connector is a command line tool deployed in Linux. In this article, we will explain the working principle of the SQL Server Hadoop connector.

Apache Hadoop Cluster

Hadoop is a master-slave architecture deployed in a cluster of Linux Hosts. To process massive data, the Hadoop environment must contain the following components:

• The master node manages slave nodes, mainly involving processing, managing, and accessing data files. When an external application sends a job request to a Hadoop environment, the master node also acts as the primary access point.
• The naming node runs the NameNode background program to manage the namespace of the Hadoop Distributed File System (HDFS) and control the access to data files. This node supports the following operations, such as opening, closing, renaming, and defining how to map data blocks. In a small environment, the named node can be deployed on the same server as the master node.
• Each slave node runs the DataNode background program to manage data file storage and process file read/write requests. The slave node is composed of standard hardware, which is relatively inexpensive and available at any time. It can run parallel operations on thousands of computers.

The relationship between components in the Hadoop environment is given. Note that the master node runs the JobTracker program, and each slave node runs the TaskTracker program. JobTracker is used to process client application requests and allocate them to different TaskTracker instances. After it receives the command from JobTracker, TaskTracker runs the assigned task together with the DataNode program and processes the data movement in each operation phase.

You must deploy the SQL Server Hadoop CONNECTOR IN THE Hadoop cluster.

MapReduce framework

As shown in, the master node supports the MapReduce framework, which relies on the Hadoop environment. In fact, you can think of Hadoop as a MapReduce framework in which JobTracker and TaskTracker play key roles.

MapReduce splits large datasets into small and manageable data blocks and distributes them to thousands of hosts. It also contains a series of mechanisms that can be used to run a large number of parallel operations, search for PB-level data, manage complex client requests, and perform in-depth analysis on data. In addition, MapReduce provides load balancing and fault tolerance functions to ensure quick and accurate operations.

The MapReduce and HDFS architectures are closely integrated. The latter stores each file as a sequence of data blocks. Data blocks are replicated across clusters. Except for the last data blocks, the size of other data blocks in the file is the same. Each slave node's DataNode program creates, deletes, and copies data blocks together with HDFS. However, an HDFS file can only be written once.

SQL Server Hadoop Connector

You need to deploy the SQL Server Hadoop connector to the master node of the Hadoop cluster. The master node also needs to install the Sqoop and Microsoft Java database connection driver. Sqoop is an open-source command line tool used to import data from a relational database, convert data using the Hadoop MapReduce framework, and re-import the data back to the database.

After the SQL Server Hadoop connector is deployed, you can use Sqoop to import and export SQL Server data. Note that Sqoop and connectors are operated in a centralized Hadoop view, this means that when you use Sqoop to import data, you retrieve data from the SQL Server database and add it to the Hadoop environment. On the contrary, exporting data refers to retrieving data from Hadoop and sending it to the SQL Server database.

Data imported and exported by Sqoop supports some storage types:

• Text Files: basic text files separated by commas;
• Sequential file: a binary file that contains serialized record data;
• Hive table: The table in Hive data warehouse, which is a special data warehouse architecture built for Hadoop.

In general, the SQL Server and Hadoop environments (MapReduce and HDFS) allow users to process massive volumes of unstructured data and integrate the data into a structured environment, create Reports and perform BI analysis.

Microsoft's Big Data Policy has just begun.

The SQL Server Hadoop connector is an important step on the way to Microsoft's big data. But at the same time, because Hadoop, Linux, and Sqoop are both open-source technologies, this means that Microsoft should be open to the open-source world on a large scale. In fact, Microsoft's plan is not just that. At the end of this year, they will also launch a solution similar to Hadoop and run it on the Windows Azure cloud platform as a service.

Next year, Microsoft plans to launch similar services for the Windows Server platform. There is no denying that the SQL Server Hadoop connector is of great significance to Microsoft. Users can handle Big Data Challenges in the SQL Server environment. I believe they will bring us more surprises in the future.