Cloud Storage

In MapReduce, shuffle is more like the inverse process of shuffling, which refers to "disrupting" the random output of the map end according to the specified rules into data with certain rules so that the reduce end can receive and process it.

Hadoop MapReduce is a programming model for data processing that is simple but powerful enough to be designed for parallel processing of big data.

Earlier we used HDFS for related operations, and we also understood the principles and mechanisms of HDFS. With a distributed file system, how do we handle files? This is the second component of Hadoop-MapReduce.

This article briefly describes the execution steps and workflow of the mapreduce programming model in the form of graphics, which is simple and easy to understand.

MapReduce is a programming model for parallel computing of large-scale data sets (greater than 1TB) to solve the computational problems of massive data.

MapReduce in Hadoop is a simple software framework based on which an application can run on a large cluster of thousands of commercial machines and process terabytes of data in parallel with a reliable fault tolerance.

Hadoop is more suitable for solving big data problems, and relies heavily on its big data storage system, namely HDFS and big data processing system. For MapReduce, we know a few questions.

Hdp hadoop installation is not a smooth success every time, inevitably will always report a lot of errors, here are some wrong solutions.

Hadoop (HDP) cluster kerberos authentication implementation, for security reasons, this article hides some system names and service names, and modified some of the parts that may cause information leakage.

"Hadoop Distributed File System (HDFS), a distributed file system that supports high-throughput access to application data；hadoop YARN, a framework for job scheduling and cluster resource management. "

Hadoop is a software framework that enables distributed processing of large amounts of data. The Hadoop distribution provides its own commercial version in addition to Apache hadoop, cloudera, hortonworks, mapR, Huawei, and DShadoop.

Currently, the Hadoop distribution has an open source version of Apache and a Hortonworks distribution (HDP Hadoop), MapR Hadoop, and so on. All of these distributions are based on Apache Hadoop.

Kafka Basic Use Method (Java). Procedure: Create topic $ cd/opt/cloudera/parcels/kafka-2.1.1-1.2.1.1.p0.18 $ bin/kafka-topics--create--zookeeper localhost : 2181--replication-factor 1--partitions 1--topic test Analysis:

Memcahced introduced, Memcached is a set of High-performance memory object caching system for some high load Web sites, the main role is to cache database query results, reduce the number of database visits to improve the response speed of dynamic Web applications. Memcached is a typical C/s architecture, so it is necessary to install server-side (Memcached) and Client (memcache). Server side is written in C language, the client can be written in any language, such as PHP, Python, per ...

The benefits of the thread pool: Reduce resource consumption: Avoid resource consumption for frequent creation and destruction of threads; increase the speed: when new tasks arrive, you do not have to create a new thread every time to execute it immediately; increase the manageability of threads: the thread pool distributes, tuned, and monitored threads uniformly.   Unrestricted creation of threads is not allowed. The realization principle of line Cheng code when the thread pool receives a new commit task, how the thread pool handles the new task, which mainly learns the thread pool's processing flow for the new task. The number of threads currently running is less than C ...

RABBIT-MQTT+PMQTT Protocol +paho Library (ubuntu16.04). Step 1 Download Install RABIIT-MQ mqtt-server on the Deb package another need sudo apt install erlang 2 boot sudo/etc/init.d/rabbitmq-server restart 3 mqtt-3.1 protocol support Rabbitmq-plugins Enable ...

Spark conversion (transform) and Action (action) list. The following func, most of the time, to make logic clearer, we recommend using anonymous functions! (lambda) "" "Ps:java and Python APIs are the same, names and parameters are unchanged." Transform meaning Map (func) Each INPUT element is exported after a Func function conversion and output an element filter (func) returns the value returned after the Func function evaluates to The input element of true is composed of ...

Docker version Jenkins built. 1. Download Docker Yum Install docker-y 1.1 configuration Docker boot from SYSTEMD systemctl enable Docker 1.2 start Docker Systemctl start Docker 2. Download Docker version Jenkins docker pull Jenkins 3. Rename Docker image, ...

Sparkha cluster configuration, spark cluster Hadoop configuration based on Hadoop HDFs. Su-rdato cd/u01 tar-zxvf spark-2.1.1-bin-hadoop2.7.tgz mv spark-2.1.1-bin-hadoop2.7 Spark #复制模版 cp/u01/spark/conf/ Spark-env.sh.template/u01/spa ...

Kubernetesscheduler Module Code learning, Scheduler module in the kubernetes is relatively easy to understand the module, but its work is more important, it is mainly responsible for those who have not found node to run the pod to select the most appropriate node. Its job is to find the right node for the pod and then submit it to apiserver Binder that the pod is already part of the node and that the Kubelet module is responsible for the subsequent work. Scheduler die ...