Python data preprocessing

I. Attribute specification

In the process of data preprocessing, if a column of data is the same or the property is the same, then this column does not help our prediction, should be removed, pandas if a column property value is the same, but there is a missing value (NaN) in this column, pandas default it has two properties, In doing this, we should first remove the missing values and then check for uniqueness. The code is as follows;

Orig_columns = Loans_2007.columnsdrop_columns = []for col in orig_columns:    col_series = Loans_2007[col].dropna (). Unique ()    if Len (col_series) = = 1:        drop_columns.append (col) loans_2007 = Loans_2007.drop (Drop_columns, Axis=1)

Second, missing value

Use the following code for missing values to see the missing values for each column in the data, for columns with more missing values, to be deleted directly, for columns with fewer missing values to delete their samples, or to populate other values instead.

Null_counts = Loans.isnull (). SUM ()

Three, character conversion

Print (Loans.dtypes.value_counts ())
OBJECT_COLUMNS_DF = Loans.select_dtypes (include=["Object"])  #选择字符型的属性

Using the above code to look at the properties of each column, Sklearn cannot handle the character type, only the numeric type is acceptable. For character types, the following can be handled:

Mapping_dict = {"    emp_length": {        "Years": Ten,        "9 years": 9,        "8 years": 8,        "7 Years": 7,        "6 yea RS ": 6,        " 5 years ": 5,"        4 Years ": 4," 3 Years ": 3," 2 years ": 2,        " 1 Year ": 1,        " < 1 year ": 0,< c13/> "N/A": 0    }}loans = loans.replace (mapping_dict) loans = Loans.drop (["Last_credit_pull_d", "Earliest_cr_line" , "Addr_state", "title"], Axis=1) loans["int_rate"] = loans["Int_rate"].str.rstrip ("%"). Astype ("float")

For can enumerate can make a dictionary, then make a replace to the data, for the column with "%", you can remove the percent sign directly.

Four, sample imbalance problem

1. Data enhancement

2, plus weight items

(1) (LR = logisticregression (class_weight= "balanced"))

(2) Set yourself, upload to Class_weight as follows:

Penalty = {    0:5,    1:1}lr = logisticregression (class_weight=penalty)

3. Multiple Model Fusion

Python data preprocessing