AC多模匹配演算法過濾敏感詞執行個體

本文章摘編、轉載需要註明來源 http://blog.csdn.net/shadowsick/article/details/8891939

在應用項目中很多時候都需要用到過濾敏感詞的功能,自己寫個遍曆明顯在小資料的時候還能湊合下,但是對於大資料的時候就有點力不從心了,這裡推薦使用ac多模匹配演算法

先來寫個應用類

/** * AC多模匹配敏感字元工具類實作類別 *  * @author shadow * @email 124010356@qq.com * @create 2012.04.28 */public class AcUtilImpl implements AcUtil {public String contrast(String filters, String word, String regex) {if (null == filters || "".equals(filters) || null == word|| "".equals(word))return "";AhoCorasick ac = new AhoCorasick();String[] strings = StringUtils.split(filters, regex);for (String string : strings)ac.add(string.getBytes(), string);ac.prepare();return matching(ac, word);}public String contrast(String[] filters, String word) {if (null == filters || filters.length <= 0 || null == word|| "".equals(word))return "";AhoCorasick ac = new AhoCorasick();for (int i = 0, len = filters.length; i < len; i++) {ac.add(filters[i].getBytes(), filters[i]);}ac.prepare();return matching(ac, word);}public String contrast(List<String> filters, String word) {if (null == filters || filters.size() <= 0 || null == word|| "".equals(word))return "";AhoCorasick ac = new AhoCorasick();for (int i = 0, len = filters.size(); i < len; i++) {ac.add(filters.get(i).getBytes(), filters.get(i));}ac.prepare();return matching(ac, word);}private String matching(AhoCorasick ac, String word) {StringBuffer buffer = new StringBuffer();Iterator<?> iterator = ac.search(word.getBytes());while (iterator.hasNext()) {SearchResult result = (SearchResult) iterator.next();buffer.append(result.getOutputs()).append(",");}return buffer.length() > 0 ? buffer.substring(0, buffer.length() - 1): "";}public static void main(String[] args) {String filters = "or,world,33,dd,test";String word = "hello world, how are you!";String regex = ",";String result = new AcUtilImpl().contrast(filters, word, regex);System.out.println(result);}}

然後運行main函數測試下,獲得的結果是

[or],[world]

這個外掛程式的效能,匹配度也灰常不錯,AhoCorasick這個類自己下載放到項目裡就可以了