Pandas converting a Chinese dataset to a numeric category data set

A machine learning contest, the main idea is as follows, this paper mainly records the process of data processing, in order to train the model, the first step is to treat the Chinese data set as a numeric category data set to save.

operator complaint and fault correlation analysis based on big data

Target: The original dataset is a table in XLS format with a large number of Chinese, with the target processed into a CSV table of numeric categories.

The original dataset is partially sliced, in the following format:

　　

The target data set is, processed into the corresponding numeric category format, as follows:

Solution: (processing Chinese requires attention to coding)

The overall two-step, 1. Extracts the value of each column (including Chinese) as the key keyword, and value is the self-increment integer that constructs the dictionary. Takes advantage of the dictionary's key unique feature.

2. Based on the previous dictionary, replace the Chinese key of the dataset with the corresponding value in the dictionary.

The source code is as follows:

From pandas import DataFrame
Import Pandas as PD
Import Matplotlib.pyplot as Plt
Import xlrd

FD = Pd.read_excel ("Complain.xls", encoding= ' Utf-8 ', Header=none)
# Fd[fd.notnull ()] #查看缺失值情况newdf=PD. DataFrame () #保存处理后的数据集keydf=PD. DataFrame () #保存中文对照表 forIinchRange (len (fd.columns)): NewData={} #利用字典的key唯一特性, remove duplicate text, and use value index=0#累加数值 Rowline=[] keylist=[] ROWDF=PD. DataFrame () forJinchRange1, Len (FD)): #1. Extract the keyword dictionary, key is Chinese, value is cumulative integer key=Fd.iloc[j,i]ifKey notinchNewdata.keys (): Newdata[key]=Index Index+=1keylist.append (key) #将中文提取出来 for easy control of ROWDF=PD. DataFrame ({i:keylist}) #保存中文对照表 KEYDF= Pd.concat ([KEYDF, ROWDF], axis=1, ignore_index=True) forKinchRange1, Len (FD)): #2according to the table, replace the Chinese with the corresponding value Rowline.append (Newdata[fd.iloc[k,i]]) ROWDF=PD. DataFrame ({i:rowline}) #保存处理后的表 NEWDF= Pd.concat ([NEWDF, ROWDF], axis=1, ignore_index=True) Print newdf.info () newdf.to_csv ('Newdata.csv') Keydf.to_csv ('Keylist.csv', encoding='GBK')

Specifically, the creation of the Pandas Dataframe table and the associated table operation, need to view the official documentation: http://pandas.pydata.org/pandas-docs/stable/merging.html

Continue to update the association rules mining and decision tree to do model processing of data.

Pandas converting a Chinese dataset to a numeric category data set