Big Data Analysis Technology

Big Data Analysis Technology

MR: offline computing framework
Storm: Real-time computing framework
Spark: memory computing framework

 

Basic architecture of Yarn

ResourceManager

? Process client requests
? Start/monitor applicationmaster
? Monitor nodemanager
? Resource allocation and Scheduling

Nodemanager

? Resource management on a Single Node
? Process commands from ResourceManager
? Process commands from applicationmaster

Applicationmaster (using mrappmaster as an example)

? Data splitting
? Apply for resources for applications and assign them to internal tasks