Maximum Entropy model document reading guide

Maximum Entropy model is a machine learning method, it has good application effects in many natural language processing fields (such as part-of-speech tagging, Chinese word segmentation, sentence boundary recognition, shallow syntax analysis, and text classification. Dr. Zhang's maximum entropy model toolkit Manual contains "further reading", which is well written. Let's put it here as a document reading guide to the maximum entropy model.
Unlike the statistical machine translation document reading guide, I am trying to learn the maximum entropy model. I have no right to speak. These documents are easy to find on Google, but most of them are relatively long (more than 30 pages), and even two are doctoral papers and more than 100 pages. I hope that beginners will not be intimidated, after all, classic things are worth further consideration!

Maximum Entropy Model tutorial reading

This section lists some recommended papers for your further reference.

1. maximum Entropy Approach to natural language processing [Berger et al ., 1996]
(required) A must read paper on applying maxent technique to natural language processing. this paper describes maxent in detail and presents an increment Feature Selection Algorithm for increasingly construct a maxent model as well as several example in statistical machine translation.

2. inducing features of Random Fields [della Pietra et al ., 1997]
(required) Another must read paper on maxent. it deals with a more general frame work: Random Fields and proposes an improved iterative Scaling Algorithm for estimating parameters of Random Fields. this paper gives theoretical background to Random Fields (and hence maxent model ). A greedy field induction method is presented to automatically construct a detail random ELDS from a set of atomic features. an word morphology application for English is developed.

3. adaptive statistical language modeling: A Maximum Entropy Approach [Rosenfeld, 1996]
This paper applied me technique to statistical language modeling task. more specically, it built a conditional maximum entropy model that inconfigurated traditional n-gram, distant n-gram and trigger pair features. significantly perplexity limit ction over baseline trigram model was reported. later, Rosenfeld and his group proposed a whole sentence exponential model that overcome the computation bottleneck of conditional me model.

4. maximum Entropy Models for natural language ambiguity resolution [ratnaparkhi, 1998]
This dissertation discussed the application of maxent model to varous natural language disambiguity tasks in detail. several problems were attacked within the me framework: Sentence boundary detection, part-of-speech tagging, shallow parsing and text categorization. comparison with other machine learning technique (Naive Bayes, Transform Based Learning, demo-tree etc .) are given.

5. The improved iterative Scaling Algorithm: a gentle introduction [Berger, 1997]
This paper describes IIS Algorithm in detail. The description is easier to understand than [della Pietra et al., 1997], which involves more mathematical notations.

6. Stochastic Attribute-value grammars (abdisney, 1997)
Abney applied improved iterative Scaling Algorithm to parameters estimation of Attribute-value grammars, which can not be corrected calculated by ERF method (though it works on pcfg ). random Fields is the model of choice here with a general Metropolis-Hasting sampling on calculating feature Expectation under newly constructed model.

7.a comparison of algorithms for maximum entropy parameter estimation [Malouf, 2003]
four iterative parameter estimation algorithms were compared on several NLP tasks. l-BFGS was observed to be the most valid tive parameter estimation method for maximum entropy model, much better than IIS and GIS. [Wallach, 2002] reported similar results on Parameter Estimation of Conditional Random Fields.

Appendix:
Dr. Zhang's maximum entropy model toolkit:
http://homepages.inf.ed.ac.uk/lzhang10/maxent_toolkit.html
for two reference pages about the maximum entropy model, the latter is also a reading list, but earlier:
1. maxent and exponential models
2.a maxent reading list