Fasttext Basic use of Java, Python as an example

Fasttext Basic use of Java, Python as an example

This morning on the subway to see someone using Fasttext for text classification, to the company tried the next situation on GitHub to find the next, the first is the C + + version of the implementation, but there are Java, Python version of the implementation, just take the test hackers,

Python case:

Python version reference, the author provides a detailed implementation, and provides the Chinese word after the data, just take down to use, thanks to the author, the code provided by the data authors are provided, point after link on the above have Baidu disk, downloadable, Java interface used to the same data:

[HTML]View PlainCopy

1. http://blog.csdn.net/lxg0807/article/details/52960072

[Python]View PlainCopy

2. Import logging
4. Import Fasttext
6. Logging.basicconfig (format='% (asctime) s:% (levelname) s:% (message) s ', Level=logging.info)
8. #classifier = fasttext.supervised ("Fasttext/news_fasttext_train.txt", "Fasttext/news_fasttext.model", Label_prefix = "__label__")
10. #load训练好的模型
12. Classifier = Fasttext.load_model (' Fasttext/news_fasttext.model.bin ', label_prefix=' __label__ ')
14. result = Classifier.test ("Fasttext/news_fasttext_test.txt")
16. Print (result.precision)
18. Print (Result.recall)
20. Labels_right = []
22. texts = []
24. With open ("Fasttext/news_fasttext_test.txt") as fr:
26. lines = Fr.readlines ()
28. For line in lines:
30. Labels_right.append (Line.split ("\ t") [1].rstrip (). Replace ("__label__", "" "))
32. Texts.append (Line.split ("T") [0])
34. # Print Labels
36. # Print Texts
38. # break
40. Labels_predict = [e[0] for e in classifier.predict (texts)] #预测输出结果为二维形式
42. # Print Labels_predict
44. Text_labels = List (set (Labels_right))
46. Text_predict_labels = List (set (LABELS_PREDICT))
48. Print (Text_predict_labels)
50. Print (Text_labels)
52. A = Dict.fromkeys (Text_labels,0) #预测正确的各个类的数目
54. B = Dict.fromkeys (Text_labels,0) #测试数据集中各个类的数目
56. C = Dict.fromkeys (Text_predict_labels,0) #预测结果中各个类的数目
58. For I in range (0,len (labels_right)):
60. B[labels_right[i]] + = 1
62. C[labels_predict[i]] + = 1
64. if labels_right[i] = = Labels_predict[i]:
66. A[labels_right[i]] + = 1
68. Print (A)
70. Print (B)
72. Print (C)
74. #计算准确率, recall rate, F value
76. For key in B:
78. p = float (A[key])/float (B[key])
80. R = Float (A[key])/float (C[key])
82. f = p * R * 2/(P + r)
84. print ("%s:\tp:%f\t%fr:\t%f"% (key,p,r,f))

Java version scenario: Githup: [HTML]View PlainCopy

1. Https://github.com/ivanhk/fastText_java

See the use of the next SH script, their own simple a text method, just use, the back will take Xgboost to compare, look at the effect, the effect can be written service to go online: [Java]View PlainCopy

2. Package test;
4. Import java.util.List;
8. Import Fasttext. Fasttext;
10. Import Fasttext. Main;
12. Import Fasttext. Pair;
16. Public class Test {
18. public static void Main (string[] args) throws Exception {
22. string[] Text = {
24. "supervised",
26. "-input",
28. "/users/shuubiasahi/documents/python/fasttext/news_fasttext_train.txt",
30. "-output", "/users/shuubiasahi/documents/faste.model", "-dim" ,
32. " Ten", "-LR", "0.1", "-wordngrams", "2", "-mincount", "1",
34. "-bucket", "10000000", "-epoch", "5", "-thread", "4"};
36. Main OP = new Main ();
38. Op.train (text);
40. Fasttext Fasttext = new Fasttext ();
42. string[] Test = { "enrolled", "Science", " student","student","student"};
44. Fasttext.loadmodel ("/users/shuubiasahi/documents/faste.model.bin");
46. List<pair<float, string>> list = fasttext.predict (test, 6); //Get the maximum possible six prediction probabilities
48. for (pair<float, string> parir:list) {
50. System.out.println ("key is:" + parir.getkey () + "value is:"
52. + Parir.getvalue ());
54. }
56. System.out.println (Math.exp (List.get (0). GetKey ())); //Get maximum predictive probability
60. }
64. }

Here the set bucket does not apply to set too large, over the General Assembly generated Oom, and the model is saved too large, the above settings model saved there are 1 G,-wordngrams can be set to 2 ratio set to 1 can improve the accuracy of the model classification, the results of the situation:

Key is:0.0 Value is:__label__edu

Key is:-17.75125 Value Is:__label__affairs

Key is:-17.75125 Value Is:__label__economic

Key is:-17.75125 Value Is:__label__ent

Key is:-17.75125 Value Is:__label__fashion

Key is:-17.75125 Value Is:__label__game

1.0

Note that Fasttext is required for the input format, the label label uses "__label__" + the actual label form, over

Contact me if you have any questions

May 2016 26 My model is on the line and it works.

Fasttext Basic use of Java, Python as an example