Stanford CoreNLP開源項目的3種編譯和運行方式

Stanford CoreNLP開源項目的3種編譯和運行方式

1.     Stanford CoreNLP簡介

Stanford CoreNLP, integrating our NER, POS tagger, and parser with a new coreference system

        官網上是如上介紹Stanford CoreNLP的。它是Stanford的NLP小組將自己的幾個關於自然語言處理的組件組合起來的一個開源項目。該工具將Stanford的NER，POS tagger，parser工具和一個新的coreference指代系統整合在一起，形成了一個完整的自然語言處理工具平台。

        如果想做自然語言處理，可以考慮使用。

        官網Link： http://nlp.stanford.edu/software/corenlp.shtml

       ：http://nlp.stanford.edu/software/stanford-corenlp-v1.1.0.tgz

2.     Stanford coreNLP的簡單使用方式

一般的java開發人員如果要使用到Stanford CoreNLP的話，一般只需要到官網上下載Stanford CoreNLP的jar包，放到項目中的classpath下即可使用。

1.下載，解壓

 

2.  eclipse下建立項目simpleNLP

3.建立lib檔案夾，將stanford CoreNLP解壓後的檔案夾中：fastutil.jar、jgraph.jar、jgrapht.jar、stanford-corenlp-2011-06-19.jar、stanford-corenlp-models-2011-06-19.jar、xom.jar這幾個jar包加到lib中。並對它們add to build path…

 

4.測試：編寫TestCoreNLP.java如下：

 

import java.util.List; import java.util.Map; import java.util.Properties;   import edu.stanford.nlp.dcoref.CorefChain; import edu.stanford.nlp.ling.CoreLabel; import edu.stanford.nlp.ling.CoreAnnotations.NamedEntityTagAnnotation; import edu.stanford.nlp.ling.CoreAnnotations.PartOfSpeechAnnotation; import edu.stanford.nlp.ling.CoreAnnotations.SentencesAnnotation; import edu.stanford.nlp.ling.CoreAnnotations.TextAnnotation; import edu.stanford.nlp.ling.CoreAnnotations.TokensAnnotation; import edu.stanford.nlp.ling.CoreAnnotations.TreeAnnotation; import edu.stanford.nlp.ling.CorefCoreAnnotations.CorefChainAnnotation; import edu.stanford.nlp.pipeline.Annotation; import edu.stanford.nlp.pipeline.StanfordCoreNLP; import edu.stanford.nlp.trees.Tree; import edu.stanford.nlp.trees.semgraph.SemanticGraph; import edu.stanford.nlp.trees.semgraph.SemanticGraphCoreAnnotations.CollapsedCCProcessedDependenciesAnnotation; import edu.stanford.nlp.util.CoreMap;   public class TestCoreNLP {          public static void main(String[] args) {                    // creates a StanfordCoreNLP object, with POS tagging, lemmatization, NER, parsing, and coreference resolution              Properties props = new Properties();              props.put("annotators", "tokenize, ssplit, pos, lemma, ner, parse, dcoref");              StanfordCoreNLP pipeline = new StanfordCoreNLP(props);                            // read some text in the text variable              String text = "Add your text here";                            // create an empty Annotation just with the given text              Annotation document = new Annotation(text);                            // run all Annotators on this text              pipeline.annotate(document);                            // these are all the sentences in this document              // a CoreMap is essentially a Map that uses class objects as keys and has values with custom types              List<CoreMap> sentences = document.get(SentencesAnnotation.class);                              for(CoreMap sentence: sentences) {                // traversing the words in the current sentence                // a CoreLabel is a CoreMap with additional token-specific methods                for (CoreLabel token: sentence.get(TokensAnnotation.class)) {                  // this is the text of the token                  String word = token.get(TextAnnotation.class);                  // this is the POS tag of the token                  String pos = token.get(PartOfSpeechAnnotation.class);                  // this is the NER label of the token                  String ne = token.get(NamedEntityTagAnnotation.class);                                         System.out.println(word+","+pos+","+ne);                }                  // this is the parse tree of the current sentence                Tree tree = sentence.get(TreeAnnotation.class);                  // this is the Stanford dependency graph of the current sentence                SemanticGraph dependencies = sentence.get(CollapsedCCProcessedDependenciesAnnotation.class);              }                // This is the coreference link graph              // Each chain stores a set of mentions that link to each other,              // along with a method for getting the most representative mention              // Both sentence and token offsets start at 1!              Map<Integer, CorefChain> graph =                document.get(CorefChainAnnotation.class);          }   }

 

這段代碼將text交給StanfordCoreNLP處理，StanfordCoreNLP的各個組件（annotator）按“tokenize, ssplit, pos, lemma, ner, parse, dcoref”順序進行處理。分別是：分詞、斷句、定詞性、詞元化、分辨具名實體、文法分析、同義字分辨等7大組件。

處理完後List<CoreMap> sentences = document.get(SentencesAnnotation.class);中包含了所有分析結果，遍曆即可獲知結果。

這裡簡單的將單詞、詞性、是否實體列印出來。

執行結果：

 

Adding annotator tokenize Adding annotator ssplit Adding annotator pos Loading POS Model [edu/stanford/nlp/models/pos-tagger/wsj3t0-18-left3words/left3words-distsim-wsj-0-18.tagger] ... Loading default properties from trained tagger edu/stanford/nlp/models/pos-tagger/wsj3t0-18-left3words/left3words-distsim-wsj-0-18.tagger Reading POS tagger model from edu/stanford/nlp/models/pos-tagger/wsj3t0-18-left3words/left3words-distsim-wsj-0-18.tagger ... done [4.4 sec]. done [4.5 sec]. Adding annotator lemma Adding annotator ner Loading classifier from edu/stanford/nlp/models/ner/all.3class.distsim.crf.ser.gz ... done [38.4 sec]. Loading classifier from edu/stanford/nlp/models/ner/muc.distsim.crf.ser.gz ... done [10.9 sec]. Loading classifier from edu/stanford/nlp/models/ner/conll.distsim.crf.ser.gz ... done [18.2 sec]. Adding annotator parse Loading parser from serialized file edu/stanford/nlp/models/lexparser/englishPCFG.ser.gz ... done [11.5 sec]. Adding annotator dcoref Add,VB,O your,PRP$,O text,NN,O here,RB,O

 

執行的時候記得設定-Xmx512m這個參數。否則提示java Heap Space…的異常。Stanford要求32位機器應該設定1800m，而64位機器應該設定3g。這個記憶體要求令人咋舌。

 

3.     項目源碼在eclipse下的編譯運行

開原始碼的使用方式其實並不是只有第2節所述的jar包使用方式。更多的時候，我們是需要將源碼下載下來，然後利用開發工具將源碼加入到自己的項目，並進行修改，然後再編譯，最後部署成一個jar包，雖然這樣做要比第2節做法複雜，但是卻可以對開源架構做有利於自己的修改。這是第2種方法做不到的。

1.下載，解壓。源碼在stanford-corenlp-2011-06-19-sources.jar中。將其中檔案解壓到src目錄。

2. eclipse中建立一個工程NLPTest，工程下建立lib目錄，將解壓出的fastutil.jar、jgraph.jar、jgrapht.jar、stanford-corenlp-models-2011-06-19.jar、xom.jar這幾個jar包放進lib中，並add to build path…

注意，這裡相比第2節的第3步少加了個stanford-corenlp-2011-06-19.jar，其實這個jar檔案就是coreNLP的源碼編譯好的class檔案打成的jar包。我們就是要把這個jar包替換成源碼。

3.將第1步解壓的src目錄下的所有原檔案拷貝到工程的src目錄下。等待eclipse的編譯完成。

4.將coreNLP原解壓檔案中的input.txt拷貝到工程根目錄下。

 

得到的工程目錄如上。

5.測試：在工程的edu.stanford.nlp.pipleline包下有一個StanfordCoreNLP.java。它是整個工程的測試類別，該目錄下還有個StanfordCoreNLP.properties，這是工程的設定檔。

只要設定好StanfordCoreNLP這個類的運行參數，整個工程就能正常運行。參數設定如下：

 

運行後會在工程根目錄下產生input.txt.xml檔案。這是所有解析結果。

如下所示：