Information extraction based on web development pattern

<span id="Label3"></p><p align="center" style="color:rgb(51,51,51)"><p align="center" style="color:rgb(51,51,51)"><strong>Information extraction based on web development pattern</strong></p></p><p align="center" style="color:rgb(51,51,51)"><p align="center" style="color:rgb(51,51,51)"><br></p></p><p style="color:rgb(51,51,51)"><p style="color:rgb(51,51,51)">Information extraction is an important part of Internet natural language processing, and the accuracy of information extraction will directly affect the subsequent Processing. The goal of the information extraction is to remove the noise and to obtain information such as the title, time, text, link, etc. of the webpage with valuable Information.</p></p><p style="color:rgb(51,51,51)"><p style="color:rgb(51,51,51)"><strong></strong></p></p><p style="color:rgb(51,51,51)"><p style="color:rgb(51,51,51)"><strong>Introduction to mainstream algorithms</strong></p></p><p style="color:rgb(51,51,51)"><p style="color:rgb(51,51,51)">There are many ways to extract web pages, for example, from algorithms: template-based, information-based, visual, semantic-based, statistical-based. From the HTML processing is divided into: based on the row block, based on the DOM Tree. Let me introduce each of Them.</p></p><p style="color:rgb(51,51,51)"><p style="color:rgb(51,51,51)">1. Based on templates, a template for URLs and HTML is generally maintained MANUALLY. When a URL is matched to a URL template, the corresponding HTML template is used to extract the information from it. This method is quick, accurate, crawl a small number of stations can be used, you can do some template Setup tool to reduce the workload, a large number of stations need more manpower maintenance template List.</p></p><p style="color:rgb(51,51,51)"><p style="color:rgb(51,51,51)">2. Based on the amount of information (the interpretation of information I will say below), see based on the line block distribution function of the body extraction, the calculation of the text in the source line on which the distribution of more, take more than the body of the line; in addition, the algorithm is based on the body density of the line calculation, simple point is the body length/ Based on the amount of information there is also another way, is to build a DOM tree, the row function into a DOM tree node evaluation Function. For the information site, this method will work very well, but need to take into account that the extraction of web information does not mean that more than the text, such as the text under a section of copyright information or site description, How to remove this information? In addition, the game download site is divided into the Game's structured information, description information, game instructions and other parts of the information is decentralized, rather than centralized, how to deal with this kind of information?</p></p><p style="color:rgb(51,51,51)"><p style="color:rgb(51,51,51)">3. Based on the vision-based page segmentation algorithm, is a specific implementation of the block-based algorithm, which is a Microsoft Research Asia algorithm for Microsoft Search engine Bing. I prefer this algorithm because it puts forward two good ideas: one is to divide the pieces according to the vision, and the other is to do the block merging according to the Vision. Based on the complexity of visual processing, the use of css, JavaScript and other engines, need to use the browser kernel library to handle html, performance may not be high. In addition, the results of this algorithm just tell you how many pieces of the page can be divided, the location of each block, the size of what, and which block and which is the body of the text needs to be further Calculated.</p></p><p style="color:rgb(51,51,51)"><p style="color:rgb(51,51,51)">4. semantic-based Text extraction, based on the anchor text and page headings and other error-prone information to find body blocks, such algorithms are effective, but still have limitations.</p></p><p style="color:rgb(51,51,51)"><p style="color:rgb(51,51,51)">5. statistical, based on the combination of block and statistics of the news text extraction and based on the similarity of web pages to remove Noise. The former use of statistics is to find the same page inside the body block, the latter is linked to the same path of different pages of the similarity to remove noise, the two have a difference. Based on statistics, you can reduce the error caused by the differences of individual web pages and improve the Accuracy.</p></p><p style="color:rgb(51,51,51)"><p style="color:rgb(51,51,51)"><strong></strong></p></p><p style="color:rgb(51,51,51)"><p style="color:rgb(51,51,51)"><strong>Look at the Web Developer's perspective</strong></p></p><p style="color:rgb(51,51,51)"><p style="color:rgb(51,51,51)">These methods, are from the law of the Web page, can solve some of the problems, and the root of the problem is that Web pages are developed by Web engineers, research their web development habits and patterns for information extraction is the most fundamental, and I have done web development, So summed up several useful patterns for information extraction:</p></p><p style="color:rgb(51,51,51)"><p style="color:rgb(51,51,51)"><strong>Mode 1 </strong>: A set of templates for similar Pages. Internet sites, broadly divided into CMS systems (such as the Imperial cms), Blog Systems (such as wordpress), Forum systems (such as discuzz), regardless of the system, the same type of Web pages are based on the same template and background data generated static or dynamic pages, the structure is the same, And the content is not the Same. If there is a revision, but also a unified revision, purely handmade pages have been very few.</p></p><p style="color:rgb(51,51,51)"><p style="color:rgb(51,51,51)"><strong>Mode 2 </strong>: information on different functions with block tags. All blocks are used block tags (group label), HTML tags have block attributes have div, TABLE, FORM, CENTER, UL, LI and so On.</p></p><p style="color:rgb(51,51,51)"><p style="color:rgb(51,51,51)"><strong>Pattern 3 </strong>: repeating structure with LOOPS. List data, forums, blog comments, usually get data rows, and then loop output according to the ROW.</p></p><p style="color:rgb(51,51,51)"><p style="color:rgb(51,51,51)"><strong>Pattern 4 </strong>: organize blocks by Information. first, there is a distinction between the style, navigation, text, related articles, comments, left navigation, the right side of the ad style are different, and replies, reply style are the Same. second, the more relevant the block, the closer the block, the text, related articles, comments are close, and the text from the right side of the advertisement is very far.</p></p><p style="color:rgb(51,51,51)"><p style="color:rgb(51,51,51)"><strong>Mode 5 </strong>: Either the web Developer level is not high, or the site comparison rogue, a lot of text is not clean, anxious ads in the grip of the Text.</p></p><p style="color:rgb(51,51,51)"><p style="color:rgb(51,51,51)">Based on the above analysis, combined with some reference algorithms above, this paper proposes an information extraction algorithm based on web development pattern, which can solve the problem of accuracy and cleanliness in information extraction. Note: <strong>accuracy</strong> refers to the integrity of the body, <strong>cleanliness</strong> refers to the body does not contain noise.</p></p><p style="color:rgb(51,51,51)"><p style="color:rgb(51,51,51)"></p></p><p style="color:rgb(51,51,51)"><p style="color:rgb(51,51,51)"><strong>The algorithm description of information extraction based on web development pattern</strong></p></p><p style="color:rgb(51,51,51)"><p style="color:rgb(51,51,51)">1. According to "mode 1" to collect the same domain name or the path of n (n>=1) pages, the same domain name or the path of the page with the same template is more likely. If n=1, then to a single page of information extraction, a single page extraction, for the news, short blog Extraction difficult, If there is a group of pages combined after the extraction can be better to solve the Problem.</p></p><p style="color:rgb(51,51,51)"><p style="color:rgb(51,51,51)">2. According to "mode 2", respectively, according to the HTML block tag to build n Dom Tree. This tree is not all HTML tags are a node, only <strong>the visual node of the block tag</strong> can establish a node, not only to meet the needs of information extraction, but also to improve efficiency. Is the 5 DOM tree Created.</p></p><p style="color:rgb(51,51,51)"><p style="color:rgb(51,51,51)"></p></p><p align="center" style="color:rgb(51,51,51)"><p align="center" style="color:rgb(51,51,51)"></p></p><p align="center" style="color:rgb(51,51,51)"><p align="center" style="color:rgb(51,51,51)">Figure 1</p></p><p style="color:rgb(51,51,51)"><p style="color:rgb(51,51,51)">3. Judge whether the N Dom is similar, mainly select the upper branches of the DOM tree to determine whether the structure is similar, take a similar DOM tree to merge its node features, perhaps N Dom belongs to more than one template, you can merge multiple Dom tree, one-off Calculation. Features are: body length, number of links, text length in links, picture size, number of Labels. Assume that the merged Dom tree is D. If the node of a path is in a different DOM tree with exactly the same characteristics, then this node can be ignored (remove duplicate noise information such as copyright, website description, etc.).</p></p><p style="color:rgb(51,51,51)"><p style="color:rgb(51,51,51)">4. According to "mode 4", a similar body block D merging, compared to 2, node 7 below 10, 11, 12 are body blocks (can be computed according to the node characteristics), with the same parent node, you can merge to Node 7. This step is mainly to have some blog or Web site, its body is distributed in several blocks, if not merged, the body of the extract will not be Complete.</p></p><p align="center" style="color:rgb(51,51,51)"><p align="center" style="color:rgb(51,51,51)"></p></p><p align="center" style="color:rgb(51,51,51)"><p align="center" style="color:rgb(51,51,51)"></p></p><p align="center" style="color:rgb(51,51,51)"><p align="center" style="color:rgb(51,51,51)">Figure 2</p></p><p style="color:rgb(51,51,51)"><p style="color:rgb(51,51,51)">5. According to "mode 3", the loop continuous block in D is merged, This category is mainly for comments, forum information, 3, nodes 2, 3, 4, 5, 6 is the same structure of nodes, merged into Node 2. If you do not merge, you will extract a small chunk of it, resulting in incomplete information. At the same time, for the loop continuous block, need to have a right to deal with, some posts and comments, the weight of comments will be larger than the blog, do not drop the right, will be drawn to comments rather than blog.</p></p><p align="center" style="color:rgb(51,51,51)"><p align="center" style="color:rgb(51,51,51)"></p></p><p align="center" style="color:rgb(51,51,51)"><p align="center" style="color:rgb(51,51,51)"></p></p><p align="center" style="color:rgb(51,51,51)"><p align="center" style="color:rgb(51,51,51)">Figure 3</p></p><p style="color:rgb(51,51,51)"><p style="color:rgb(51,51,51)">6. Find the most informative block. Here's an explanation of the concept of information, which <strong>is a quantitative measure of how text, links, pictures, videos, animations, and their styles are communicated to users</strong> . Plainly speaking, is the Web page want to give users what information, content page to the user is content, and the navigation page to the user is the link, the amount of information calculation formula is not the Same. Figure 4 is a Web page structure, the root node 1 has 3 nodes: 2, 3, 4; According to the information calculation formula, the node 3 information is the largest, the node 3; node 3 has 7, 82 nodes, 7 max; 7 below is 11, so node 11 is the body Node. Why not take node 15, there are two possibilities, one is that node 15 has been merged into node 11 in the 4th step, and the other is that the node 15 information account for node 11 of the amount of information is too small, will not be selected.</p></p><p style="color:rgb(51,51,51)"><p style="color:rgb(51,51,51)"><em>Information Formula = Body Information + link information + picture information + video (including flash) info + tag Information</em></p></p><p align="center" style="color:rgb(51,51,51)"><p align="center" style="color:rgb(51,51,51)"></p></p><p align="center" style="color:rgb(51,51,51)"><p align="center" style="color:rgb(51,51,51)"></p></p><p align="center" style="color:rgb(51,51,51)"><p align="center" style="color:rgb(51,51,51)">Figure 4</p></p><p style="color:rgb(51,51,51)"><p style="color:rgb(51,51,51)">7. According to "mode 5", the body node is found only that the body is contained in the body node, found in the body node still contains noise, such as the body block of the inclusion of advertising information, such as the body block contains too many related links information and so On. At this time, the body blocks need to be further cleaned to eliminate noise information. For the forum, if only the information of the post itself, and not user information, can be repeated according to the characteristics of the forum replies, the amount of information on each block in each reply block variance, large variance for the post block (because the length of the post varies greatly), the variance is small user information block (user information block difference).</p></p><p style="color:rgb(51,51,51)"><p style="color:rgb(51,51,51)"><br></p></p><p style="color:rgb(51,51,51)"><p style="color:rgb(51,51,51)"><strong>Algorithm Advantage Analysis</strong></p></p><p style="color:rgb(51,51,51)"><p style="color:rgb(51,51,51)">1. Using group tags and tags in the CSS to block, instead of VIPs color, size, location and other information, simplifying the calculation process, high efficiency, so the calculation, in the actual application of the effect is also good. If the color, position, size factors can be taken into account, will further improve the accuracy Rate. Whether you need to deal with css, see the actual needs.</p></p><p style="color:rgb(51,51,51)"><p style="color:rgb(51,51,51)">2. Using the same template under different page structure similarity, and the similarity of the loop block in the page, to extract information, than a single page, a single block of information extraction, the accuracy of the increase in the rate of more than 3% (estimated value). For example, The news (only a word) is more difficult to deal with, such as only the content of the posts in the Forum (left personal information, signature files do not), such as extracting posts and not comments, etc., <strong>by observing a set of similar structures to process information</strong> , this idea can be extended to other types of page information Extraction.</p></p><p style="color:rgb(51,51,51)"><p style="color:rgb(51,51,51)">3. The algorithm is more general, only need to be based on different algorithms to merge similar blocks and design a reasonable amount of information formula, can provide various types of application scenarios for the extraction of data, such as extracting text, images and video, links and structured content.</p></p><p style="color:rgb(51,51,51)"><p style="color:rgb(51,51,51)">4. Further optimization, on the basis of structural similarity, the characteristics of the <strong>Web page structure and Web page information template</strong> can be retained, in case the page can not be extracted information, especially for forums and blogs, such as the number of replies to the page is more Important.</p></p><p style="color:rgb(51,51,51)"><p style="color:rgb(51,51,51)"><br></p></p><p style="color:rgb(51,51,51)"><p style="color:rgb(51,51,51)"><strong>Practical Application Effect</strong></p></p><p style="color:rgb(51,51,51)"><p style="color:rgb(51,51,51)">In practical application, the text extraction part, for tens of thousands of sites (including information, blogs, Forum Site) Data sampling to detect, accurate rate can reach more than 96%.</p></p><p style="color:rgb(51,51,51)"><p style="color:rgb(51,51,51)">This algorithm is used for the link Analysis section of the hub, analyzing the page links that need to be crawled in the hub page, not including the popular, navigation and other links on the left and right sides, and the tens of thousands of hub page tests with a accuracy rate of 92%. -if you take the position of the block into consideration, the effect will be better.</p></p><p style="color:rgb(51,51,51)"><p style="color:rgb(51,51,51)"><br></p></p><p style="color:rgb(51,51,51)"><p style="color:rgb(51,51,51)"><strong>Some other issues when extracting information</strong></p></p><p style="color:rgb(51,51,51)"><p style="color:rgb(51,51,51)">Label fault tolerance: This algorithm does not recognize the content of attribute, does not recognize the content of CSS and script, only need to deal with the label matching, even if the label matching error does not matter, as long as the information can be Extracted.<br>Code recognition: The charset can be extracted from the header, and if not, it can be automatically identified with Mozilla's charset detection Component. Coding recommendations are converted to utf-8.</p></p><p style="color:rgb(51,51,51)"><p style="color:rgb(51,51,51)">Language Recognition: It is possible to use Utf-8 's coding interval to determine the probability of the distribution of characters in the language range.</p></p><p style="color:rgb(51,51,51)"><p style="color:rgb(51,51,51)">Title extraction and Purification: anchor text and title combination, according to the rule truncation title, the "_ News Center _ sina" and other meaningless removal, can also be based on similar pages of the title of the common part of the removal to Intercept.</p></p><p style="color:rgb(51,51,51)"><p style="color:rgb(51,51,51)">Date Time recognition: The body area is not far from the place, with the regular to Match. If you have more than one time, you can take more than a certain time (2000 later?). The time closest to the present but not exceeding the current Time.</p></p><p style="color:rgb(51,51,51)"><p style="color:rgb(51,51,51)">Image Extraction: Extract the large picture link in the body area, the text of the picture can be extracted from the text below the picture or the text around the picture and the text of the Title.</p></p><p style="color:rgb(51,51,51)"><p style="color:rgb(51,51,51)">Link Extraction: Extract the most linked block, if the link + introduction + thumbnail of the hub page, you can put the text and the image as a weight in the Calculation. Hub page is also a variety of forms, the difficulty is not smaller than the text Extraction.</p></p><p style="color:rgb(51,51,51)"><p style="color:rgb(51,51,51)"></p></p><p style="color:rgb(51,51,51)"><p style="color:rgb(51,51,51)">Others: later thought to add<br></p></p><p style="color:rgb(51,51,51)"><p style="color:rgb(51,51,51)"></p></p><p style="color:rgb(51,51,51)"><p style="color:rgb(51,51,51)">Appendix: The following is a document section, shared, for Reference.</p></p><p style="color:rgb(51,51,51)"><p style="color:rgb(51,51,51)">Information extraction based on web development pattern</p></p><p style="color:rgb(51,51,51)"><p style="color:rgb(51,51,51)">Web Page Information Extractor</p></p><p><p>Information extraction based on web development pattern</p></p></span>