Data Cleaning 4

1. Read The data:

1.1 If The data is not in. csv file. We have to search for the special Read method

All_survey = Pandas.read_csv ("Schools/survey_all.txt", delimiter= "\ T", encoding= ' windows-1252 ') # Read/ http kunststube.net/encoding/ for the introduction of encoding.

1.2 read a big set of data, so we is using the For loop to Read through the data.

　For F in Data_files:
File = Pd.read_csv ("schools/{0}". Format (f)) #When It related to a variable in the "", we can not directly use the Vari Able name in the string.
f = F.replace (". csv", "")
DATA[F] = File

　　1.3 Combine Some dataframe into one by using concat () function.

Survey = Pd.concat ([all_survey,d75_survey],axis = 0) #

2. Cleaning up the data:

　　In the combined Dataframe, it's inavoidable to has lots of ' NaN ' inside. So we need to deal with these "NaN"

2.1 We need to figure out which column is relevant. And extract them from the original Dataframe.

2.2 Some of the column name may has different column name but shows the same content. We need to change them into one.

2.3 To unify the string, we can add,minus, change, numeric the column names

3. Filting the data:

3.1 We can use FindAll () and Split function to extract certain string we need from the whole string.

　　def extract_lat (data):
Lat_lon = Re.findall ("\ (. +,.+\)", data)
lat = Re.split (",", Lat_lon[0])
Final_lat = Lat[0].replace ("(", "" ")
Return Final_lat

data["Hs_directory" ["lat"] = data["Hs_directory" ["Location 1"].apply (Extract_lat) #loop through each row of the DataFrame in certain column to call the function.

　　3.2 Find the relevant dataset from each column. and store them into another Dataframe.

4.

Data Cleaning 4