Big Data architect Skills Atlas

Big Data Universal processing platform

1. Spark
2. Flink
3. Hadoop

Distributed storage

Hdfs

Resource Scheduling

Yarn

Mesos

Machine learning Tools

Mahout

1. Spark Mlib
2. TensorFlow (Google Department)
3. Amazon Machine Learning
4. DMTK (Microsoft Distributed Machine Learning tool)

Data analysis/Data Warehouse (SQL Class)

1. Pig
2. Hive
3. Kylin
4. Spark SQL,
5. Spark DataFrame
6. Impala
7. Phoenix
8. ELK

8.1 ElasticSearch

8.2Logstash

8.3Kibana

Message Queuing

1. Kafka (Pure log class, high throughput)
2. Rocketmq
3. ZeroMQ
4. ActiveMQ
5. RabbitMQ

Flow-based computing

1. Storm/jstorm
2. Spark Streaming
3. Flink

Log Collection

Scribe

Flume

Programming languages

1. Java
2. Python
3. R
4. Ruby
5. Scala

Data analysis and mining

Matlab

Spss

Sas

Visualization of data

1. R
2. D3.js
3. Echarts
4. Excle
5. Python

Machine Learning

Machine Learning Basics

1. Clustering
2. Time series
3. Recommendation system
4. Regression analysis
5. Text mining
6. Decision Tree
7. Support Vector Machine
8. Bayesian classification
9. Neural network

Machine learning Tools

1. Mahout
2. Spark Mlib
3. TensorFlow (Google Department)
4. Amazon Machine Learning
5. DMTK (Microsoft Distributed Machine Learning tool)

algorithm

Consistency

1. Paxos
2. Raft
3. Gossip

Data

1. Stacks, queues, linked lists
2. Hash table
3. Binary tree, red black tree, B-Tree
4. Figure

Common Algorithms

1. Sorting

Insert Sort

Bucket sort

Heap Sort

2. Quick Sort

3, maximum sub-array

4. Longest common sub-sequence

5. Minimum spanning tree

Shortest path

6. Storage and operation of matrices

Cloud Computing

Cloud Services

1. Saas
2. Paas
3. Iaas
4. Openstack
5. Docker

End.

Transferred from: http://www.36dsj.com/archives/4520

Source: http://www.ha97.com/5734.html

Big Data architect Skills Atlas