Summary of data processing methods in the beginner's competition (python)

For the first time, Tianchi Big Data Contest (blood sugar prediction), ranked 1% in the preliminaries. Because I'm not familiar with Python, take a look at some of the python methods used in the game (more basic details, big guy Detour):

1. Data exploration

Data.info () Data.describe ()

Using the above two lines of code, you can initially see the entire data distribution, missing and so on

2. The presence of gender in the data is a string representation, using the map method, digitizing him, and of course using Onehot. (python turns string data into numbers)

data[' sex ' = data[' sex '].map ({' Male ': 1, ' Female ': 0})

3. The date in the data processing, there are two, one is directly using get_dummies (), or if you are using the LGB model can directly use parameters Categorical_feature

The other is to calculate the difference between the two dates: (Python calculates the difference of the date)

Import timeimport datetime# Date Standardization data[' physical examination Date ' = Pd.to_datetime (data[' Checkup date '],format= '%d/%m/%y ') days =[]for day in data[' Physical examination Date ']:      days.append ((Day-pd.to_datetime (' 2017-09-15 ')). Days) data[' physical examination date '] = Day

In the game, there are big guys open up their own code, the processing of time is:

data[' physical examination Date ' = (Pd.to_datetime (data[' Physical examination Date ')-Parse (' 2017-10-09 ')). Dt.days

　　However, some of the things I did in this way found that his recognition of time was not very good, such as the data if it was 02-11-2017 he would be identified as February 11, 2017, but the 30-10-2017 also identified October 30, 2017. This is the problem, so use the previous method, the format to specify, there is no such a problem

4.pandas Dropna () removes too many columns with empty values. In the data game, there is a large amount of data are Nan, this time if too many Nan features or samples can be deleted:

Import pandas as Pddata.dropna (axis =1, thresh = 100,inplace = True)

Here the parameter explanation: Axis=1 means to delete the column, if it is 0 delete the row, thresh=100 means that if the null value is more than 100 to delete the column, Inplace=true represents the change itself. There are also parameters how specific can be referenced in the documentation: official documents

5. Use Pandas (Python) to read the file containing Chinese characters, the data feature names used in the contest are all English, because it is a medical term, it is more familiar to see Chinese, so when reading, you need to specify the format:

Import Pandas as Pddata = Pd.read_csv (' .../data.csv ', encoding = ' gb2312 ')

Use encoding to specify the format of the document, if you do not know what the document is, or if you want to convert the format, you can use notepad++ open to convert the format.

6. Use the time package to record when the algorithm runs. Because of the time efficiency of the algorithm, the time package can be used to calculate the Times:

Import Timeimport datetimeprint (' Start training ... ') T0 = Time.time () ... Algorithm ... print (' Training time {} seconds. Format (Time.time ()-T0))

This will give you the time to run the algorithm.

First write here, for data visualization processing, there are many many ways, the details are not recorded. The following items, especially model comparisons and visualizations, are very helpful, and can be made up when they are empty.

6. Feature expansion

7. Main formation Analysis dimensionality reduction

8. Normalization

9. Model Data Comparison

9.1 K-Fold Model Prediction Data graph (graph of predicted and real values)

9.2 K-Fold Model Prediction Data graph (graph of predicted and true values of fitting training data)

Normal distribution (method and code) for comparison results of more than 9.3 models

XGB Feature Selection

11. Three major correlation coefficients (Pearson, Spearman, Kendall) feature selection

Summary of data processing methods in the beginner's competition (python)