C # JAVA operating Hadoop (HDFS, Map/Reduce) Real process overview. Download components and source code. Unsolved: Response status code does not indicate success: 500 .,

C # JAVA operating Hadoop (HDFS, Map/Reduce) Real process overview. Download components and source code. Unsolved: Response status code does not indicate success: 500 .,

1. Hadoop environment Configuration Overview

Three virtual machines: Ubuntu 16.04.

Hadoop version: 2.7.2

NameNode: 192.168.72.132

DataNode: 192.168.72.135, 192.168.72.136

Note: The configuration process is not described. There are many online versions.

Ii. eclipse (JAVA) Environment Configuration Overview

Operating System: Windows 10

Eclipse version: Mars.2 Release (4.5.2)

1. hadoop-eclipse-plugin-2.7.2.jar components put plugins directory, specific configuration to find online.

2. Configure HADOOP_HOME environment variable: E: \ hadoop-2.7.2, configure PATH environment variable, add: % HADOOP_HOME % \ bin

3. Download: winutils.exe and Hadoop. dll files, put in E: \ hadoop-2.7.2 \ bin; Hadoop. dll put in Windows \ System32 download.

4. REFERENCE The hadoop package. For correctness, share \ hadoop \ common, share \ hadoop \ hdfs, share \ hadoop \ mapreduce, share \ hadoop \ yarn, and all jar packages in lib under each directory are referenced.

5. Create a project wordcount and copy a lot of code online.

6. Sub-branch effect:

StackTrace "in System. threading. tasks. task. throwIfExceptional (Boolean includeTaskCanceledExceptions) \ r \ n in System. threading. tasks. task. wait (Int32 millisecondsTimeout, CancellationToken cancellationToken) \ r \ n in System. threading. tasks. task. wait () \ r \ n in Microsoft. hadoop. webClient. webHCatClient. webHcatMapReduceStreamingExecutor. execute (Boolean throwOnError) \ r \ n in Microsoft. hadoop. mapReduce. execution. hadoop. streamingJobExecutorBase. executeCore (Type mapper, Type CER, Type combiner, HadoopJobConfiguration config) \ r \ n in Microsoft. hadoop. mapReduce. execution. hadoop. streamingJobExecutorBase. execute (Type mapperType, Type reducerType, Type combinerType, HadoopJobConfiguration config) \ r \ n in Microsoft. hadoop. mapReduce. execution. hadoop. streamingJobExecutorBase. execute [TMapper, TReducer] (HadoopJobConfiguration config) \ r \ n in Hadoop_Demo.Program.Main (String [] args) Position C: \ Users \ WXZZ \ Desktop \ Hadoop_Demo \ Program. cs: Row 61 "string

This problem has never been resolved after I searched the network. For a reply, see:

If the Hdfs protocol is incorrect, the StorageSystem operation will not be successful. However, if mapreduce execution fails, I doubt whether the webhcat configuration is incorrect or the request data protocol itself is faulty.

Iv. Summary

It's just a small experiment and there is no real production environment for applications. I have been using C # for development. This problem cannot be solved and is helpless. Make a summary first and study it later.

5. Download components and source code

Http://pan.baidu.com/s/1i48Ln4D

 

1. [serialization] C # communication (Serial Port and network) Framework Design and Implementation

2. [Open Source] C # cross-platform Iot communication framework ServerSuperIO (SSIO) Introduction

2. Overall system construction solution using SuperIO and open-source cross-platform Iot framework ServerSuperIO

3. C # technical route of industrial IoT and integrated system solutions (data sources, data collection, data upload and receiving, ActiveMQ, Mongodb, WebApi, and mobile App)

5. ServerSuperIO Open Source Address: https://github.com/wxzz/ServerSuperIO

Internet of Things & integrated technology (. NET) QQ Group:54256083