pandas 讀取大檔案 read_table C-engine CParserError: Error tokenizing data

標籤：exce   erer   style   color   getting   complete   strong   loop   encoding   

解決辦法：

pd_data = pd.read_table(comment_file,header=None,encoding=‘utf-8‘, engine=‘python‘)

官網解析：

　　　 engine : {‘c’, ‘python’}, optional

Parser engine to use. The C engine is faster while the python engine is currently more feature-complete.

 

1、

iterator : boolean, default False

Return TextFileReader object for iteration or getting chunks with get_chunk().

或者通過chunk 擷取

pd_data = pd.read_table(comment_file,header=None,encoding=‘utf-8‘,iterator=True)
# print(pd_data)
# pd_data_t = pd.read_table(comment_file,header=None,encoding=‘utf-8‘, engine=‘python‘)
# return;
loop = True
chunk_data = []
chunk_size = 1024
while loop:
   try:
      pd_data_tmp = pd_data.get_chunk(chunk_size)
      chunk_data.append(pd_data_tmp)
   except StopIteration:
      loop = False
df = pd.concat(chunk_data,ignore_index=True)

 

pandas 讀取大檔案 read_table C-engine CParserError: Error tokenizing data