Compile an HTML file analysis program using Java

Use Java to compile the HTML file analysis program-general Linux technology-Linux programming and kernel information. The following is a detailed description. Abstract: This article focuses on the application of StreamTokenizer in the Java language in compiling HTML file analysis programs, and introduces the routines for downloading web pages in bytes.

I. Overview

The core of the Web server is to make a correct analysis of the tags in the HTML file. The interpreter in a programming language also analyzes the reserved words in the source file and then explains them. In this case, you need to separate the tags in the HTML file to find the desired file name and directory. Before the emergence of Java, similar work needs to analyze each character in the file to find the required part, not only a large amount of programming, but also prone to errors. In my recent project, I used the Java input stream class StreamTokenizer to analyze HTML files, which has better results. Here, we need to download the HTML file from a known Web page, analyze it, and then download the HTML file contained in this page (if it is in the Frame), image files, and Class (Java Applet) files.

Ii. StreamTokenizer class

StreamTokenizer: the token-based input stream is used to convert an input stream into a token stream. There are three types of token entities in the token stream: Word (Multi-character token), single-character token, and blank (including the description statements in Java and C/C ++ ).

The constructor of the StreamTokenizer class is StreamTokenizer (InputStream in)

This class has some public instance variables: ttype, sval, and nval, indicating the token type, the current string value, and the current numeric value, respectively. When we need to obtain characters between tokens (that is, tags in HTML), We Should access the variable sval. The method for reading the next token is to call nextToken (). The returned value of method nextToken () is int type. There are four possible responses:

StreamTokenizer. TT_NUMBER: indicates that the read token is a number, and the number value is double. It can be read from the instance variable nval.

StreamTokenizer. TT_WORD: indicates that the read token is a non-numeric word (other characters are also included). Words can be read from the instance variable sval.

StreamTokenizer. TT_EOL: indicates that the read token is the row Terminator.

If you have read to the end of the stream, nextToken () returns TT_EOF.

Before calling nextToken (), you must set the syntax table of the input stream so that the analyzer can recognize different characters. The WhitespaceChars (int low, int hi) method defines the range of meaningless characters. The WordChars (int low, int hi) method defines the character range of the constructed word.

3. Program Implementation

1. Implementation of the HtmlTokenizer class

Before analyzing a token stream, you should first set the syntax table of the token stream. In this example, let the program tell which word is the HTML Tag. The following section defines the token stream class for the HTML Tag we need. It is a subclass of StreamTokenizer:


Import java. io .*;
Import java. lang. String;
Class HtmlTokenizer extends
StreamTokenizer {
// Define each tag. The tag here is only required in this example,
Scalable as needed
Static int HTML_TEXT =-1;
Static int HTML_UNKNOWN =-2;
Static int HTML_EOF =-3;
Static int HTML_IMAGE =-4;
Static int HTML_FRAME =-5;
Static int HTML_BACKGROUND =-6;
Static int HTML_APPLET =-7;

Boolean outsideTag = true; // determines whether the tag is in progress.

// Constructor, which defines the syntax table of the token stream.
Public HtmlTokenizer (BufferedReader r ){
Super (r );
This. resetSyntax (); // reset the syntax table
This. wordChars (0,255); // the token range is all characters
This. ordinaryChar ('<'); // delimiter used to mark both sides of the HTML
This. ordinaryChar ('> ');
} // End of constructor

Public int nextHtml (){
Int token; // token
Try {
Switch (token = this. nextToken ()){
Case StreamTokenizer. TT_EOF:
// If you have read to the end of the stream, TT_EOF is returned.
Return HTML_EOF;
Case '<': // enter the tag field
OutsideTag = false;
Return nextHtml ();
Case '>': // output tag field
OutsideTag = true;
Return nextHtml ();
Case StreamTokenizer. TT_WORD:
// If the current token is a word, determine the tag
If (allWhite (sval ))
Return nextHtml (); // filter Spaces
Else if (sval. toUpperCase (). indexOf ("FRAME ")
! =-1 &&! OutsideTag) // mark the FRAME
Return HTML_FRAME;
Else if (sval. toUpperCase (). indexOf ("IMG ")
! =-1 &&! OutsideTag) // mark IMG
Return HTML_IMAGE;
Else if (sval. toUpperCase (). indexOf ("BACKGROUND ")
! =-1 &&! OutsideTag) // mark the BACKGROUND
Return HTML_BACKGROUND;
Else if (sval. toUpperCase (). indexOf ("APPLET ")
! =-1 &&! OutsideTag) // tag the APPLET
Return HTML_APPLET;
Default:
System. out. println ("Unknown tag:" + token );
Return HTML_UNKNOWN;
} // End of case
} Catch (IOException e ){
System. out. println ("Error:" + e. getMessage ());}
Return HTML_UNKNOWN;
} // End of nextHtml

Protected boolean allWhite (String s) {// filter all spaces
// Implementation
} // End of allWhite

} // End of class

The above method was tested by the author in a recent project. The operating system is Windows NT4, and the programming tool uses Inprise Jbuilder3.