Spark Dataframe API Finishing

1, create the dataframe from the list
Each element of the list is converted to a row object, and the Parallelize () function converts the list to the RDD,TODF () function to convert the RDD to Dataframe
From Pyspark.sql import Row
L=[row (name= ' Jack ', age=10), Row (Name= ' Lucy ', age=12)]
Df=sc.parallelize (L). TODF ()

There is no schema for creating the data in the Dataframe:rdd from the Rdd, using row to create the Schema,spark and then determining the data type based on the sampled data, if not defined, the system will give none (a bit of a pit)
l=[(' Jack ', Ten), (' Lucy ', 12)]
Rdd=sc.parallelize (L)
Person=row (' name ', ' age ')

Df=rdd.map (Lambda R:person (*r)). TODF ()

Create dataframe using type description from Rdd

From Pyspark.sql. Types Import *

Schema=structtype ([Structfield (' Name ', StringType (), False)

, Structfield (' Age ', Integertype (), False)]

Df=sqlcontext.applyschema (Rdd,schema)

This method will strictly check the data type, if the data does not conform to the definition format of the error, when Rdd is empty, still able to successfully create Dataframe

2, register temporary table
Df.registertemptable ("T1"), but the T1 table is only valid under the current SqlContext
Df_table=sqlcontext.tables () command to query all table reports, temporary tables, hive tables
=> Table_name=sqlcontext.tables ()
=> T1=table_name.filter (table_name.istemporary)
3, the use of aggregate functions
Can be combined with agg (), pyspark.sql.function to handle
From Pyspark.sql import functions as F
=> L=[row (name= ' Jack ', age=10), Row (Name= ' Lucy ', age=12)]
=> df=sc.parallelize (L). TODF ()
=> Max_age=df.agg (F.max (Df.age)). A () [0]
4,groupby Group Aggregation
L=[row (strategy= ' zhld ', code= ', amount=100), row (strategy= ' zhld ', code= ', amount=200 '), row (strategy= ' Dxzl '), Code= ' ", amount=1000)]
Df=sc.parallelize (L). TODF ()
Code_count=df.groupby (Df.strategy). Agg (F.count (Df.code). Alias (' num ')
=>[row (strategy= ' zhld ', num=2), Row (strategy= ' Dxzl ', num=1)]

5, data curing to memory in several ways
If the data needs to be used frequently, want to base the table, but not quite so, can be continuously updated to memory
Cache Dataframe:df.cache ()
Merging of 6,dataframe partitions
Dataframe is always converted to RDD for calculation, dataframe partition is RDD partition
DF.COALESCE (1) DF's number of partitions merged to 1
Df.rdd.getNumPartitions () returns the number of partitions for DF
7, the deletion of the column, some cases need the associated table, this time to introduce a new column, you can use drop, remove the newly introduced column
Df=df.drop (Df.name)
8, go to Heavy dropdumplicates ()
L=[row (strategy= ' zhld ', code= ', amount=100), row (strategy= ' zhld ', code= ', amount=200 '), row (strategy= ' Dxzl '), Code= ' ", amount=1000)]
Df=sc.parallelize (L). TODF ()
Df2=df.dropdumplicates (Df.strategy)
9,fileter None Value Filter, filter () Accept true| False
Df=df.filter (Df.name.isNull ())
Df=df.filter (Df.name.isNotNull ())
10, to seek the intersec of the set and to find the difference set subtract
L=[row (strategy= ' zhld ', code= ', amount=100), row (strategy= ' zhld ', code= ', amount=200 '), row (strategy= ' Dxzl '), Code= ' ", amount=1000)]
Df=sc.parallelize (L). TODF ()
Df1=df.filter (df.strategy== ' zhld ')
Df2=df.intersect (DF1)
Df3=df.subtract (DF1)
11,replace replacement
A=sc.parallelize ([Alice, ' Beijing '), (' John ', ', ' Shanghai ')]
Age=row (' name1 ', ' age ', ' addr ')
A=a.map (Lambda r:age (*r)). TODF ()
C=a.replace (' Alice ', ' Stefan ', name1)
C=a.replace ([' Alice ', ' John '],[' Stefan ', ' Jace '],name)
12,show () View Dataframe data situation
Functions that can be supplied using data types in 13,select
Cosmetic between (): A=ch.select (Ch.name,ch.age.between (3,5). Alias (' flag '). Show () return true| False
String EndsWith (): B=a.select (A.name,a.name.endswith (' n '))
The Substr:a=ch.select of the string (Ch.age,ch.name.substr (2,3). Alias (' name1 ') starts with the second character for three characters
F.when A=ch.select (Ch.name,f.when (ch.age>3,1). When (ch.age<=3,-1). otherwise (0). Alias (' flag ')
    F provided regular expressions: A=temp.select (f.regexp_extract (Temp.str, ' (\d+)-(\d+) ', 1). Alias (' d ')