Hadoop for. NET developers__.net

Hadoop for. NET Developers (i): Understanding Hadoop

Over the years, big data has become an exciting source of analysis for the industry. For the purpose of this blog series, I will loosely define this term to refer to the expansion of data sources from the data core business system-the category of traditional business intelligence-including new data sources, either historically overlooked or new available.

This is a rough simplification of the big data, but it is inherently the challenge of working with the new data platform that has been pushed through new data platforms. When our attention is in Hadoop, the broadest endorsement of these new data platforms, the large data of such a limited definition seems to suffice.

Hadoop is an elastic, distributed, modeless data-processing platform that is ideal for you to have a large number of datasets, with a single record containing a small number of values, such as log files, because it provides a low input for data access solutions. This is also a good platform for complex analysis and interpretation of complex data, such as XML or JSON documents, image files, and/or may be subject to variable interpretations such as customer tweets (in JSON files).

In addition, Hadoop is an excellent platform when you need massive scalability beyond the traditional relational database platform. Having said that, I did not find this final solution is applicable to many of my clients (although it is applicable to some). For my work and for the vast majority of people, Hadoop's flexibility and economics are often the most compelling reasons to explore the platform. Hadoop for. NET Developers (ii): Infrastructure

Hadoop is an implementation of a set of interrelated project components. The core components are mapreduce, used to process job execution, and a storage layer that is typically implemented as a Hadoop Distributed File System (HDFS). For the purpose of this article, we will assume that HDFs is in use.

The components of Hadoop are implemented through a series of servers called data (or compute) nodes. These nodes are where the data is stored and processed.

The name of the node server remains in the environment of the data node, whose data is stored on which node on the track, and provides the data to the node as a single entity. This singular representation is called a cluster. If you are familiar with the RDBMS implementation terminology cluster, please note that there is not necessarily any shared storage or other resources between nodes. The Hadoop cluster is pure logic.