Comprehensive in-depth analysis of spark2--knowledge points, source code, Tuning, JVM, graph calculation, project

Source: Internet
Author: User
Tags class operator shuffle

Comprehensive in-depth analysis of spark2--knowledge points, source code, Tuning, JVM, graph calculation, project
Course View Address: http://www.xuetuwuyou.com/course/220
The course out of self-study, worry-free network: http://www.xuetuwuyou.com
Lecturer: Watermelon Teacher


A total of 14 chapters, 316 sections, the course from spark-related technical aspects of all-round analysis, and finally the actual project: User interactive behavior Analysis System, DMP User Portrait system, to spark to do a comprehensive application to explain, can say a set in hand, play all the invincible hand!


1th Chapter: Scala
Task 1:java and Scala comparison
Task 2: Why learn Scala
Task 3:scala Compiler installation
Task 4: First Scala program written
Task 5:scala Tool Installation
Task 6: Programming with Idea
Task 7:idea hit Jar pack
Task 8: Declaration of a variable
Task 9:scala Data type
Task 10:if Expression
Task 11: Code block
Task 12: Cyclic-while
Task 13: Cyclic-for
Task 14:scala operator
Task 15: Definition of a method
Task 16: Defining functions
Task 17: Decorative Design
Task 18:java to explain functional programming
Task 19: Knowledge Review
Task 20: Fixed-length arrays and edge-length arrays
Task 21: Conversion and traversal of arrays
Task 22: An array of commonly used algorithms
Task 23:map Collection
Task 24: Tuple operations
Task 25:list Collection Operations
Task 26:scala Implement Word Count
Task 27:set Collection Operations
Task 28:lazy Features
Task 29:scala Course Description
Task 30: Definition of class
Task 31: View the fixed class file
Task 32: Primary and secondary constructors
Task 33: Morning Knowledge Review
Task 34: Objects
Task 35:apply method
Task 36:trait (trait)
Task 37: Extend the application
Task 38: Inheritance
Task 39: Abstract class
Task 40: Pattern Matching
Task 41:scala String Printing
Task 42: Sample Class
Task 43:option (Some,none)
Task 44: Partial function
Task 45: Closures
Task 46:curring
Task 47: Hermit parameters
Task 48: Hermit Conversion
Task 49: The Hermit Conversion Opportunity 2 case Demo
Task 50: Hermit conversion Case 1
Task 51: Hermit conversion Case 2
Task 52: Upper and lower bounds
Task 53: Upper Bound
Task 54: Nether Case
Task 55: View Boundaries
Task 56: Co-change
Task 57: Contravariance
Task 58: Knowledge Summary
Task 59:socket Job
Task 60: Job Requirements analysis
Task 61: Job Code implementation
Task 62: About actor Knowledge description
Basic concept explanation of Task 63:actor
Task 64:actor Case Demo
Task 65: Case two requirements analysis
Task 66: Case Code Demo (top)
Task 67: Case Code demo (bottom)

The 2nd Chapter: Sparkcore
Task 68: How to learn about open source technology
Task 69: What is spark
Four characteristics of Task 70:spark
Task 71:4spark fast Use (top)
Task 72:spark fast Use (bottom)
Task 73: What is an RDD
Task 74: Demonstrate what an RDD is
Task 75:spark The running process of the task
Task 76:9hadoop Cluster Construction
Task 77:spark Cluster Construction
Task 78:sparkha Cluster Construction
Task 79:scala developing Spark program Demo
Task 80:java7 developing SPARK programs
Task 81:java8 developing SPARK programs
Task 82:idea How to play Maven package
Task 83: Submit a task to the spark cluster
How task 84:rdd are created
Task 85: Description of the spark script
Task 86:transformation and Action principles
Task 87: Broadcast variables
Task 88: Accumulating variables
Task 89: Share variables using demo
Task 90: Persist
Task 91:checkpoint
Task 92: Additional Notes on persistence
Task 93:standalone Run mode
Task 94:spark-on-yarn
Task 95:spark-on-yarn Principle Description
Task 96:historyserver Service Configuration
Task 97:map-flatmap-filter
Task 98:sortbykey-reducebykey
Task 99:join-union-cogroup
Task 100:intersection-distinct-cartes
Task 101:mappartitions-repartition-coal
Task 102:coalesce and repartition Difference supplement
Task 103:aggregatebykey-mappartitionswi
Task 104: Description of the action operator
Task 105: Description of the Collect operator
Task 106:spark two-time ordering
Task 107: Narrow dependencies and wide dependencies
Task 108: Examples of narrow dependencies and wide dependencies
Task 109: noun Interpretation
Task 110:stage Partitioning algorithm
Scheduling of Task 111:spark tasks

3rd: Spark Tuning
Task 112: Avoid creating a duplicate Rdd
Task 113: Reuse the same rdd as much as possible
Task 114: Persist a multiple-use RDD
Task 115: Try to avoid using the shuffle class operator
Task 116: Shuffle operations with map-side pre-aggregation
Task 117: Using high-performance operators
Task 118: Broadcast Large variables
Task 119: Optimize serialization performance with Kryo
Task 120: Optimizing Data structures
Task 121: Data localization
Task 122: The principle of data skew and how to position data skew
Task 123: Preprocess Data using Hive ETL
Task 124: Filter A handful of keys that cause tilt
Task 125: Increase the degree of parallelism of shuffle operations
Task 126: Two-phase aggregation (local aggregation + global aggregation)
Task 127: Convert reduce join to map join
Task 128: Sample tilt key and split join operation
Task 129: Join with the random prefix and the expansion rdd
Task 130: Comprehensive Application of solutions
Task 131: Various shuffle versions
Task 132:shuffle Tuning
Task 133:spark Resource Tuning
Task 134:spark 1.5 version memory model
Memory model for Task 135:spark two
Task 136:whole-stagecodegeneration

4th Chapter: JVM Tuning
Schema of the task 137:JVM
Task 138: How the three regions work together
Task 139: Heap Structure
Task 140:JDK eight memory model
Task 141: Heap Memory Overflow Case Demo
Task 142:ma Tool Brief Introduction
Task 143:GC Log Format description
Task 144: Heap Memory Configuration Demo
Task 145: Stack parameter configuration
Task 146: Garbage Collection Algorithm Introduction
Task 147:stop-the-world
Task 148: Garbage Collection algorithm
Task 149: Introduction to the garbage collector
Task 150: Common collector configuration Demo
Task 151:cms garbage collector
Task 152:HADOOPJVM Tuning Demo
Task 153: Introduction to the garbage collector
Task 154: Introduction to Performance monitoring tools
Task 155: Large objects go straight into the old age

The 5th chapter: Sparkcore Source Code Analysis
Task 156: How to find the source code
Task 157: How to associate the source code
Task 158:master START process
Task 159:master and worker start-up process
Task 160:sparak-submit Submission Process
Task 161:sparkcontext Initialization
Task 162: Create a TaskScheduler
Task 163:DAGSCHEDUELR Initialization
Task 164:taskschedulerimp Start
Task 165:master Resource Scheduling algorithm
Task 166:TASKSCHEDULERIMLUML diagram
Task 167:executor Registration
Start UML diagram for Task 168:executor
Task 169:spark Task Submission
Task 170:task Task Run
Task 171:spark Task Submission Detail process
Task 172:spark Task submission Process drawing summary
Mission 173:blockmanager in-depth analysis
Mission 174:cachemanager in-depth analysis

The 6th chapter: Sparksql
Task 175: Description of the default number of partitions
Mission 176:sparkcore Official Case Demo
Mission 177:spark's Past life
Release Notes for Task 178:spark
Task 179: What is Dataframe
Task 180:dataframe First Experience
Task 181:rdd turn Dataframe mode one
Task 182:rdd converted to Dataframe Mode II
Task 183:rdd VS DataFrame
-load of Task 184:sparksql data source
-save of Task 185:sparksql data source
JSON and parquet of task 186:sparksql data sources
JDBC of task 187:sparksql data source
Task 188:spark hive for the data source
Task 189:thriftserver
Task 190:sparksql Case Demo
Task 191:sparksql and Hive integration
UDF of Task 192:sparksql
UDAF of Mission 193:sparksql
window function of Task 194:sparksql
Task 195:goupby and Agg
Task 196: Knowledge Summary

The 7th chapter: Kafka
Task 197: Why Kafka appear
The core concept of task 198:kafka
Task 199:kafka core concept again carding
Task 200: Introduction to various languages
Task 201: The benefits of the messaging system
Task 202: Classification of the message system and (Pull,push) differences
Task 203:kafka The architecture of the cluster
Construction of Task 204:kafka cluster
Task 205: Cluster Test Demo
Ha for Task 206:kafka data
The design of Task 207:kafka
Task 208:kafak Code Test
Task 209: Jobs
Offset of Task 210:kafka

The 8th chapter: sparkstreaming
Task 211: Brief talk about the future of sparkstreaming
Running process of Task 212:sparkstreaming
Task 213:dstream Drawing detailed
Task 214: Flow of streaming calculations
Task 215:socketstreaming Case Demo
Task 216:hdfsdstream Case Demo
Task 217:updatestatebykey Case Demo
Task 218:transform blacklist filter Demo
Task 219:window Action Case Demo
Task 220:transform blacklist filter Demo Supplement
Task 221:foreachrdd Case Demo
Task 222:kafka-sparkstreaming Integration Demo
Task 223:kafka Multi-threaded consumption data
Task 224:kafka consuming data in parallel using the thread pool

The 9th chapter: Streaming Tuning
Fault tolerance of task 225:sparkstreaming
Task 226:sparkstreaming VS Storm
Task 227:sparkstremiang and Kafka Integration (manual control offset
The degree of parallelism of task 228:sparkstreaming tuning
Task 229:sparkstreaming Tuning the memory
The serialization of Task 230:sparkstreaming tuning
The JVM&GC of Task 231:sparkstreaming tuning
Task 232:sparkstreaming Tuning individual task runs slowly
Resource instability of Task 233:sparkstreaming tuning
Task 234:sparkstreaming Data volume explosion

The 10th chapter: Streaming Source code
Mission 235:1sparkstreaming Source Guide Introduction
Task 236:sparkstreaming Operating principle
The principle of task 237:sparkstreaming communication model
Initialization of Task 238:stremaingcontext
Task 239:receiver START Process Guide
Task 240:receiver START Process UML Summary
Principle analysis of Task 241:block generation
Analysis of task 242:block generation and storage principle
Task 243: Responsibility chain Mode
Task 244:blockrdd Build and Job task submission
Task 245:blockrdd Build and Job Task submission Summary

The 11th chapter: SPARKGRAPHX
Task 246: Graph Calculation Introduction
Task 247: Figure Calculation Case Demo
Task 248: Basic Composition of graphs
Task 249: Figure Storage
Task 250: Find a friend case demo

The 12th chapter: SPARK2VSSPARK1
Task 251:spark new features
Task 252:rdd&dataframe&dataset
Task 253:rdd&dataframe&dataset
Task 254:sparksession Access Hive Supplemental Instructions
Task 255:dataframe and Datasetapi merging

The 13th Chapter: Integrated Project: User interactive behavior Analysis system
Task 256: Introduction to the project process
Task 257: Overall overview of the project
Task 258: Data sources for big data projects
Task 259: Project Background
Task 260: Common Concepts
Task 261: Project Requirements
Task 262: Project Consolidation process
Task 263: Thinking raised from the design of the table
Task 264: Get task parameters
Task 265: Requirements for a data message
Task 266: Requirements filter sessions based on criteria
Task 267: An illustrative example of demand
Task 268: Demand One-click order Payment category TOPN (top)
Task 269: Demand One-click order Payment category TOPN (bottom)
Task 270: Demand analysis of two requirements
Task 271: Requirements two data information
Task 272: Demand Two get user behavior data
Task 273: Requirement two user table and information table join
Task 274: Demand analysis of second demand
Task 275: Requirements two custom UDF functions
Task 276: Requirements Two-custom UDAF function
Task 277: Demand statistics on the number of product clicks in each region
Task 278: Demand two city information table and commodity information table join
Task 279: Demand two regions hot commodity statistics
Task 280: Requirements Two results Persistence Guide database
Task 281: Two Summary of requirements
Task 282: Demand analysis of three requirements
Task 283: Requirements three data information
Task 284: Three ways to comb the demand
Task 285: Requirements Three get data from Kafka
Task 286: Three requirements for blacklist filtering of data
Task 287: Demand three dynamic blacklist (top)
Task 288: Demand three dynamic blacklist (next)
Task 289: Demand three real-time statistics every city in the provinces advertising click
Task 290: Demand three real-time statistics of the provinces traffic click
Task 291: Demand three real-time statistics ads click Trend
Task 292: Three Summary of requirements

The 14th chapter: DMP User Portrait system
Task 293: Project Background
Task 294:DSP Process
Task 295: Project Process description
Task 296:utils Tool class development
Task 297: Demand-function development
Task 298: Package commits the code to the cluster to run
Task 299: Requirements II Description
Task 300: Reporting Requirements description
Task 301: Statistical data distribution in various provinces and cities
Task 302: Define a Word table statistic function
Task 303: Provincial City Report statistics
Task 304:app Report Statistics
Task 305: User Portrait requirements
Task 306: Tag
Task 307: Merging Context labels
Task 308: Context label test run
Task 309: Why do we need figure calculation
Task 310: Basic Concepts of graphs
Task 311: Simple Case Demo
Task 312: Thinking of merging context tags
Task 313: Simple Case Demo Description
Task 314: Continue to comb your ideas
Task 315: Generate a user relationship table
Task 316: Merge labels





Learn the hassle-free web Spark Course Summary:

Spark from beginner to practical
Course View Address: http://www.xuetuwuyou.com/my/course/186

Big data analytics based on Python spark
Course View Address: http://www.xuetuwuyou.com/course/173

Spark full explaining (based on SPARK2 version + Spark tuning + Hyper-case)
Course View Address: http://www.xuetuwuyou.com/course/149

Spark+kafka real-time streaming machine learning combat
Course View Address: http://www.xuetuwuyou.com/course/147

In-depth spark machine learning combat (user behavior analysis)
Course View Address: http://www.xuetuwuyou.com/course/144


Comprehensive in-depth analysis of spark2--knowledge points, source code, Tuning, JVM, graph calculation, project
Course View Address: http://www.xuetuwuyou.com/course/220

DMP User Portrait System (SPARKGRAPHX sparkcore Sparksql)
Course View Address: http://www.xuetuwuyou.com/course/221

Comprehensive in-depth analysis of spark2--knowledge points, source code, Tuning, JVM, graph calculation, project

Contact Us

The content source of this page is from Internet, which doesn't represent Alibaba Cloud's opinion; products and services mentioned on that page don't have any relationship with Alibaba Cloud. If the content of the page makes you feel confusing, please write us an email, we will handle the problem within 5 days after receiving your email.

If you find any instances of plagiarism from the community, please send an email to: info-contact@alibabacloud.com and provide relevant evidence. A staff member will contact you within 5 working days.

A Free Trial That Lets You Build Big!

Start building with 50+ products and up to 12 months usage for Elastic Compute Service

  • Sales Support

    1 on 1 presale consultation

  • After-Sales Support

    24/7 Technical Support 6 Free Tickets per Quarter Faster Response

  • Alibaba Cloud offers highly flexible support services tailored to meet your exact needs.