Conditional Random Fields: a beginner's survey

Conditional Random Fields: A Beginner's Survey

One interesting project that I am involved in these days involves certain problems in Intelligent Tutors. It turns out that perhaps one of the best ways to tackle them is by using Conditional
Random Fields (CRFs). Many attempts to solving these problems still involve Hidden
Markov Models (HMMs). Since I have never really been a Graphical Models guy (though I am
Always fascinated) so I found the going on studying CRFs quite difficult. Now that the survey is more or less over, here are my suggestions for beginners to go about learning them.

Tutorials and
Theory

1. Log-Linear Models and Conditional Random Fields (Tutorial by Charles
Elkan)

Log-linear Models and Conditional Random Fields
Charles
Elkan

6
Videos: Click on Image above to view

Two directions of approaching CRFs are especially useful to get a good perspective on their use. one of these is considering CRFs as an alternate to Hidden Markov Models (HMMs) while another is to think of CRFs building over Logistic Regression.

This tutorial makes an approach from the second ction and is easily one of the most basic around. most people interested in CRFs wowould ofcourse be familiar with ideas of maximum likelihood, logistic regression etc. this tutorial does a good job, starting
With the absolute basics-talking about logistic regression (for a two class problem) to a more general multi-label machine learning problem with a structured output (outputs having a structure ). I tried reading a few tutorials before this one, but found
This to be the most comprehensive and the best place to start. It however seems that there is one lecture missing in this series which (going by the Notes) covered more training algorithms.

2. survey papers on Relational Learning

These are not really tutorials on CRFs, but talk of sequential learning in general. for beginners, these surveys are useful to clarify the range of problems in which CRFs might be useful while also discussing other methods for the same briefly. I wocould recommend
These two tutorials to help put CRFs in perspective in the broader machine learning sub-area of relational learning.

-Machine Learning for sequential learning: A Survey (Thomas
Dietterich)

PDF

This is a very broad survey that talks of sequential learning, defines the problem and some of the most used methods.

-An Introduction to Structured Discriminative Learning (R Memisevic)

PS

This tutorial is like the above, however focuses more on comparing CRFs with large margin methods such as SVM. Giving yet another interesting perspective in placing CRFs.

3. Comprehensive CRF Tutorial (Andrew
McCallum and Charles
Sutton)

PDF

This tutorial is the most compendious tutorial available for CRF. while it claims to start from the bare bone basics, I found it hard for a start and took it on third (after the above two ). it is potentially the starting and ending point for a more advanced Graphical
Models student. it is extensive (90 pages) and gives a feeling of comfort with CRFs when done. it is definitely the best tutorial available though by no means the most easiest point to start if you have never done any sequential learning before.

This might be considered an extension to this tutorial by McCallum et al: CRFs
For Relational Learning (PDF)

4. Original CRF Paper (John
Lafferty et al .)

PDF

Though not necessary to learn CRFs given extends better tutorials, this paper is still recommended, being the first on CRFs.

5. Derivations (Rahul Gupta)

PDF

This report is good for one to go through the detailed derivations of the equations in CRF.

___________________

Extensions
To the CRF concept

There are a number of extensions to CRFs. The two that I have found most helpful in my work are (these are easy to follow given the above ):

1. Hidden
State Conditional Random Fields (h crf)

2. Latent
Dynamic Conditional Random Fields (LDCRF)

Both of these extensions work to include hidden variables in the CRF framework.

___________________

Software Packages

1. Kevin Murphy's CRF toolbox (MATLAB)

2. MALLET (I
Haven'tused MALLET, it is Java based)

3. HCRF-LDCRF Library (MATLAB,
C ++, Python). As the name suggests, this package is for HCRF and LDCRF, though can be used as a standalone package for CRF as well.