Sensitive word Filtering Algorithm

String
Multi-Mode exact match (dirty words/sensitive words/keyword filtering algorithm)-practical F mode of ttmp Algorithm

String
Multi-Mode exact match (dirty word/sensitive word search algorithm) -- An overview of the B mode of ttmp Algorithm

String
Multi-Mode exact match (dirty word/sensitive word search algorithm) algorithm Prepass II

String
Multi-Mode exact match (dirty word/sensitive word search algorithm)

 

String
Multi-Mode exact match (dirty word/sensitive word search algorithm)-the theory of ttmp algorithm is as follows:

Use DFA for text filtering

DFA and text filtering

Text filtering is an essential feature for large websites, and is required by many text websites. So how to design
An efficient text filtering system is very important.

Brief description of text filtering requirements: Determine which subsets of set a belong to Set B. For javaeye, if the user sends
For a table article (set a), we need to determine whether some keywords in this article belong to Set B, and B is generally a list of prohibited words.

If you don't know how to use contains or regular expressions, you may think about these methods.
Methods do not work. The only better algorithm is DFA.

I. dfa introduction:

Those who have learned how to compile must know that in the lexical analysis phase, the set of text in the source code into the syntax is implemented by determining finite automatic machines. However, DFA is not only used in lexical analysis.
It is not limited to the computer field.

The basic function of DFA is to get the next state through the event and the current state, that is, event + state = nextstate,

Let's look at a state chart that can be found everywhere:

Uppercase letters are state, while lowercase letters are action. We can see S + A = u, u + A = Q, S + B = V, and so on. In general, we can use a matrix to represent the entire state transition process:

---------------

Status/character a B

S U V

U q v

V U q

Q

However, the state chart can have many data structures. The matrix above is just a simple example for ease of understanding. Next, the text filter system mentioned in this article will use another data structure for automatic
Machine Model

2. Text Filtering

In a text filtering system, to cope with high concurrency, it is important to reduce the number of computations as much as possible. In DFA, there is basically no computing, some are status transfer. While
Constructing a list of prohibited texts into a state machine is troublesome to implement using matrices. The following describes a simple implementation method, that is, the tree structure.

All prohibited words are essentially composed of ASCII codes, and the text to be filtered is essentially a set of ASCII codes, for example:

Input is a = [101,102,105, 112,110,]

List of prohibited words:

[102,105]

[98,112]

Then our task is to construct the above two prohibited words into a DFA, so that the input a can achieve the function of searching for prohibited words through the transfer on this DFA.

Tree Structure: The basic method for implementing the DFA is the relationship between the index of the array and the array value (in double array trie, it is also based on this basic method)

So 102 can actually be seen as an array index, while 105 is an index in the next array pointed to by the index 102. If there is no value after the index 105, it indicates that the banned word is over.

In this way, you can construct a DFA tree structure.

Next, traverse each byte in the input text, and then perform status transfer in DFA to determine whether a prohibited word exists in the input text.

1. Public class DFA {
2. Private string [] arr = {"DFA", "disgusting", "da "};
3. Private node rootnode = new node ('R ');
5. Private string content = "Hello DFA world DFA, haha! Disgusting ";
7. Private list <string> words = new arraylist <string> ();
   
8. Private list <string> word = new arraylist <string> ();
   
9. Int A = 0;
11. Private void searchword (){
12. Char [] chars = content. tochararray ();
13. Node node = rootnode;
14. While (A <chars. Length ){
15. Node = findnode (node, Chars [a]);
16. If (node = NULL ){
17. Node = rootnode;
18. A = A-word. Size ();
19. Word. Clear ();
20. } Else if (node. Flag = 1 ){
21. Word. Add (string. valueof (chars [a]);
22. Stringbuffer sb = new stringbuffer ();
23. For (string STR: Word ){
24. SB. append (STR );
25. }
26. Words. Add (sb. tostring ());
27. A = A-word. Size () + 1;
28. Word. Clear ();
29. Node = rootnode;
30. } Else {
31. Word. Add (string. valueof (chars [a]);
32. }
33. A ++;
34. }
35. }
37. Private void createtree (){
38. For (string STR: ARR ){
39. Char [] chars = Str. tochararray ();
40. If (chars. length> 0)
41. Insertnode (rootnode, chars, 0 );
42. }
43. }
45. Private void insertnode (node, char [] CS, int index ){
46. Node n = findnode (node, CS [Index]);
47. If (n = NULL ){
48. N = new node (CS [Index]);
49. Node. nodes. Add (N );
50. }
52. If (Index = (CS. Length-1 ))
53. N. Flag = 1;
55. Index ++;
56. If (index <CS. length)
57. Insertnode (n, Cs, index );
58. }
60. Private node findnode (node, char c ){
61. List <node> nodes = node. nodes;
62. Node Rn = NULL;
63. For (node N: nodes ){
64. If (n. c = c ){
65. Rn = N;
66. Break;
67. }
68. }
69. Return rn;
70. }
72. Public static void main (string [] ARGs ){
73. DFA = new DFA ();
74. DFA. createtree ();
75. DFA. searchword ();
76. System. Out. println (DFA. Words );
77. }
80. Private Static class node {
81. Public char C;
82. Public int flag; // 1: indicates the end, 0: Continuation
83. Public list <node> nodes = new arraylist <node> ();
    
85. Public node (char c ){
86. This. C = C;
87. This. Flag = 0;
88. }
90. Public node (char C, int flag ){
91. This. C = C;
92. This. Flag = flag;
93. }
94. }
95. }