Python read in document form

Summary of the previous code.

Python3 Method 1: The document is saved in the. json format in the folder, the folder is only a single layer, the folder in the Chinese file to be read (application see 20170525-052501.py)

1Path = R'C:\Users\user\Desktop\heilongjiang.dbw.cn\\'2Documents = []3num =04      forFileinchos.listdir (path) [0:]:5File = path +file6         if  notFile.endswith ('. JSON'):Continue7f = open (file,'RT', encoding='Utf-8')8data =F.read ()9BODY =json.loads (data)Ten  OneContent = body['content'] A         #Print (content) -num + = 1 -         Print(Num,' ', file) theDocuments.append (Fenci (content))

Python2 Method 2: Folders are also available under folders (with category directories), and multi-category document reads. (see 20170530-corpus_segment.py for application)

1Corpus_path ="train_corpus_small/"  #Non-word classification corpus path2Seg_path ="train_corpus_seg/"  #Classification Corpus path after word segmentation4Catelist = Os.listdir (Corpus_path)#get all subdirectories under Corpus_path5 6     #get all the files under each directory7      forMydirinchOs.listdir (corpus_path):8Class_path = Corpus_path + Mydir +"/"  #to spell the path of a taxonomy subdirectory9Seg_dir = Seg_path + Mydir +"/"  #Corpus classification after spelling out participleTen         if  notOs.path.exists (Seg_dir):#whether the directory exists, if not created One os.makedirs (Seg_dir) AFile_list = Os.listdir (Class_path)#get all the files under Class_path -          forFile_pathinchFile_list:#traverse the category directory file -FullName = Class_path + File_path#spell out file name full path theContent = ReadFile (fullname). Strip ()#Read File contents -Content = Content.replace ("\ r \ n","")#Remove line breaks and extra spaces \ r \ n -Content_seg = Jieba.cut (Content.strip ())#Word breaker for file contents -SaveFile (Seg_dir + File_path," ". Join (CONTENT_SEG))#Save the processed file after the word breaker corpus

Solutions for reading encoding problems in Python2: (see 20170530-test3.py for applications)

1Corpus = []2Tfidfdict = {}3      forFile_pathinchOs.listdir (start_path):4FullName = Start_path +File_path5Content =ReadFile (FullName). Strip ()6         #content = Content.decode (' GBK ', ' ignore '). Encode (' Utf-8 ')7Content = Content.replace ("\ r \ n","")8         content = Content.decode ('utf-8') 9Content_seg =Jieba.cut (Content.strip ())TenSaveFile (End_path + File_path," ". Join (CONTENT_SEG)) One      forFile_pathinchOs.listdir (end_path): AFullName = End_path +File_path -Content =ReadFile (FullName). Strip () -         content = Content.decode ('utf-8')  theContent = Content.replace ("\ r \ n",""). Strip () - corpus.append (content) -         #Print Corpus -      forIinchRange (len (corpus)): +         PrintSTR (corpus[i])

There is one more question: Why write Reload (SYS) before sys.setdefaultencoding

1 Import SYS 2 Reload (SYS) 3 sys.setdefaultencoding ('UTF8')

Reference Document: Http://www.360doc.com/content/15/0105/15/9934052_438371998.shtml

Studying the module loading process of Python, after Sys is loaded, the Setdefaultencoding method is removed, so we are going to set the system code by re-importing Sys.

Python read in document form