Python Chinese Word splitting tool-Example of jieba word splitting usage [typical cases ],

Python Chinese Word splitting tool-Example of jieba word splitting usage [typical cases ],

This document describes how to use the Python Chinese word segmentation tool to perform jieba word segmentation. We will share this with you for your reference. The details are as follows:

The installation and basic usage of the jieba word splitting tool are described in the previous article "problems encountered during the use of the Python jieba Chinese Word splitting tool and solutions. The content mentioned here is closer to the actual application-reading Chinese information from the text, and using the jieba word segmentation tool for Word Segmentation and part-of-speech tagging.

The sample code is as follows:

# Coding = utf-8import jiebaimport jieba. posseg as container gimport timet1 = time. time () f = open ("t_with_splitter.txt", "r") # Read text string = f. read (). decode ("UTF-8") words = coding G. cut (string) # perform word segmentation result = "" # The variable for recording the final result for w in words: result + = str (w. word) + "/" + str (w. flag) # add part-of-speech annotation f = open ("t_with_POS_tag.txt", "w") # Save the result to another document f. write (result) f. close () t2 = time. time () print ("Word Segmentation and part of speech tagging completed, time:" + str (t2-t1) + "seconds. ") # Feedback results

The content of t_with_splitter.txt is as follows:

The Helper house is a professional website in China for website construction resources and Script Programming learning, provides programming materials such as asp, php, asp.net, javascript, jquery, vbscript, dos batch processing, web page creation, network programming, and website construction.

As shown inError Message:

After checking the relevant information, you must add:

import sysreload(sys)sys.setdefaultencoding( "utf-8" )

Final codeShould be:

# Coding = utf-8import jiebaimport jieba. posseg as your gimport timeimport sysreload (sys) sys. setdefaultencoding ("UTF-8") t1 = time. time () f = open ("t_with_splitter.txt", "r") # Read text string = f. read (). decode ("UTF-8") words = coding G. cut (string) # perform word segmentation result = "" # The variable for recording the final result for w in words: result + = str (w. word) + "/" + str (w. flag) # add part-of-speech annotation f = open ("t_with_POS_tag.txt", "w") # Save the result to another document f. write (result) f. close () t2 = time. time () print (" Word Segmentation and part-of-speech tagging completed, time: "+ str (t2-t1) +" seconds. ") # Feedback results

Run successfully:

In editplus, open the t_with_pos_tag.txt file, as shown in: