[Java Primary]:22. Network programming

Basic Concepts

Computer network: The distribution of computers in different geographical areas and specialized external equipment with a communication line interconnection into a large-scale, powerful network system, so that many computers can easily communicate with each other information, sharing hardware, software, data information and other resources.

The main functions of computer network:

Resource sharing;

Information transmission and centralized processing;

Balanced load and distribution processing;

Integrated Information Services (www/Integrated Services Digital Network ISDN).

Network Communication Protocol: Communication protocols must be agreed on in the computer network, such as the rate, transmission code, code structure, Transmission control steps, error control and other standards.

Network communication interface: In order to enable dialogue between the two nodes, a communication tool (the interface) must be established between them so that information can be exchanged between each other. The interface consists of two parts:

Hardware Device: Realizes information transmission between nodes.

Software device: A contractual agreement that provides for communication between the parties

Network tiering

Layering reason: Because of the complex links between nodes, in the formulation of the agreement, the complex components are decomposed into some simple components, and then compound them up. The most commonly used composite method is the hierarchical way, that is, the same layer can communicate, the previous layer can call the next layer, and the next layer does not occur.

Package/Split

web crawler

Web crawler (also known as Web spider, Network robot, in the middle of the foaf community, more often called the Web Chaser), is a certain rules, automatically crawl the World Wide Web information program or script. Other infrequently used names are ants, auto-indexing, simulation programs, or worms.

With the rapid development of the network, the World Wide Web becomes the carrier of a lot of information, how to extract and use this information effectively becomes a huge challenge. Search engine, such as the traditional general search engine altavista,yahoo! and Google, as a tool to help people to retrieve information as a user access to the World Wide Web portal and guide. However, these universal search engine also has some limitations, such as:

(1) Users of different fields and backgrounds often have different search purposes and requirements, and the results returned by general search engine contain a lot of web pages that users don't care about.

(2) The goal of general search engine is to maximize the network coverage, the contradiction between limited search engine server resources and unlimited network data resources will deepen further.

(3) The rich data form of the World Wide Web and the continuous development of network technology, pictures, databases, audio, video multimedia and other different data, the general search engine is often the content of these information dense and have a certain structure of the data can not be very good to find and access.

(4) General search engine mostly provides keyword-based retrieval, which makes it difficult to support queries based on semantic information.

In order to solve the above problems, the focus crawler of the related Web resources was born. The Focus crawler is a program that automatically downloads Web pages and, depending on the intended crawl target, has the option of accessing the Web pages on the World Wide Web with relevant links to get the information you need. Unlike general-purpose crawlers, focus crawlers do not seek large coverage, but aim to crawl Web pages related to a particular topic content and prepare data resources for topic-oriented user queries.

1. Focus crawler working principle and key Technology overview

Web crawler is a program that automatically extracts Web pages, and it is an important component of search engine to download Web pages from the World Wide Web. The traditional crawler starts from the URL of one or several initial web pages, obtains the URL on the initial page, and in the process of crawling the Web page, continuously extracts the new URL from the current page into the queue until it satisfies the system's stop condition. The work flow of the focus crawler is more complex, and it is necessary to filter the links that are irrelevant to the topic according to a certain page analysis algorithm, and keep the useful links and put them into the queue of URLs waiting to be crawled. It then selects the next page URL to crawl from the queue according to a certain search strategy, and repeats the process until a certain condition of the system is reached. In addition, all crawled Web pages will be stored by the system, for certain analysis, filtering, and indexing, so that after the query and retrieval, for the focus of the crawler, the results of this process may also provide feedback and guidance for the subsequent crawl process.

The focus crawler also needs to solve three major problems relative to general-purpose web crawlers:

(1) Description or definition of the grab target;

(2) Analysis and filtering of web pages or data;

(3) A search strategy for URLs.

The description and definition of the grab target is the basis for determining how the Web page analysis algorithm and the URL search strategy are developed. and the Web page analysis algorithm and the candidate URL sorting algorithm are the key to determine the service form and crawl behavior of the Crawler Web page that the search engine provides. These two-part algorithms are closely related.

2. Fetch Target Description

The existing focus crawler can be classified into target-based web features, target-based data patterns and domain-based concepts in 3 ways.

The objects crawled, stored and indexed by crawlers based on the features of the target Web page are generally Web sites or Web pages. According to the seed sample acquisition method can be divided into:

(1) A pre-given initial fetch seed sample;

(2) Pre-given page categories and catalogs corresponding to the classification of the seed samples, such as the Yahoo! classification structure;

(3) The sample of grab target determined by user behavior is divided into:

(a) The capture sample showing the callout during the user's browsing;

(b) Access patterns and related samples are obtained through user log mining.

Among them, the webpage characteristic can be the content characteristic of the webpage, also can be the link structure characteristic of the webpage, and so on.

Demo

<span style= "FONT-SIZE:18PX;" ><span style= "FONT-SIZE:18PX;" >package com.bjsxt.wsl;//Hellospider class import Java.io.bufferedreader;import Java.io.ioexception;import Java.io.inputstream;import Java.io.inputstreamreader;import Java.net.url;import java.net.URLConnection; public class Hellospider {publicstatic void main (string[] args) throws IOException {for (int i = 0; I < 10;                            i++) {//declaration address Urlurl = new URL ("http://www.iteye.com/news/31449");                            Crawling data disguised as a browser urlconnectionconnection = Url.openconnection (); Connection.setrequestproperty ("User-agent", "mozilla/5.0" (Windows NT 6.1;                            WOW64) applewebkit/537.36 (khtml, Likegecko) chrome/48.0.2564.116 safari/537.36 ");                            Crawl Information Inputstreaminputstream = Connection.getinputstream ();        View data in a stream (character stream)                    Bufferedreaderbufferedreader = new BufferedReader (new InputStreamReader (InputStream, "UTF-8"));                            Start reading data Stringbufferbuffer = new StringBuffer ();                            Stringline = "";                            while (line = Bufferedreader.readline ())! = null) {buffer.append (line);                   } System.out.println (buffer); }}}</span></span>

UDP Programming

In the existing network, there are two main ways of network communication:

1. TCP (Transmission Control Protocol) mode

2, UDP (User Datagram Protocol) mode

To facilitate understanding of these two approaches, let's look at an example first. When you use your phone, there are two ways to send messages to someone: make a phone call and text a message. By making a call, you can guarantee that the information will be passed on to someone else, because it is confirmed that the message was received when someone answered the call. The way to send text messages is inexpensive and easy to use, but recipients may not receive them.

In network communication, the TCP method is similar to the call, the use of this method of network communication, the need to establish a dedicated virtual connection, and then reliable data transmission, if it fails, the client will automatically resend the data. And the UDP way is similar to sends the text message, uses this way carries on the network communication, does not need to establish the specialized virtual connection, the transmission is not very reliable, if sends the failure the client cannot obtain.

Both of these transmission methods are used in the actual network programming, the important data generally use the TCP data transmission, and a large number of non-core data are transmitted through UDP, in some programs and even use these two methods for data transmission.

Because TCP needs to establish a dedicated virtual connection and verify that the transmission is correct, TCP is a bit slower to use, and the amount of data that is transmitted is slightly larger than UDP.

In the Java API, the implementation of UDP programming, including client network programming and server-side network programming, mainly implemented by two classes, namely:

(1) The Datagramsocket class implements network connections, including client network connections and server-side network connections. Although the UDP mode of network communication does not need to establish a dedicated network connection, but after all, it is necessary to send and receive data, Datagramsocket implementation is the transmitter when sending data, and the role of the listener when receiving data. Analogous to a network connection in TCP, this class can be used either to implement client connections or to implement server-side connections.

(2) The Datagrampacket class implements encapsulation of the data that is transmitted in the network, that is, the object of the class represents the data exchanged in the network. In UDP network programming, whether the data to be sent or the data to be received must be processed into an object of type Datagrampacket, which contains the address sent to, the port number sent to, and the content sent. In fact, the role of the Datagrampacket class is similar to the real letter, in the letter contains the address of the letter sent to the recipient, as well as the content sent, the post office only need to follow the address to pass. When receiving data, the received data must also be processed into an object of type Datagrampacket, which contains information such as the sender's address, port number, and the contents of the data. Compared with the TCP network transmission, IO programming is not necessary in the network programming of UDP, and the structure is simpler than the TCP mode of network programming.

The demo function is to implement the client program's system time to the server side, after the server receives the time, the feedback string "OK" to the client. The client code that implements the feature is as follows:

<span style= "FONT-SIZE:18PX;" ><span style= "FONT-SIZE:18PX;" >package UDP; Import java.net.*;import java.util.*;/** * Simple UDP client, implement system time function to server side */public class Simpleudpclient {public STA tic void Main (string[] args) {datagramsocket ds = null;//Connection object Datagrampacket SE NDDP; Send packet object Datagrampacket RECEIVEDP; Receive Packet object String serverhost = "127.0.0.1"; server IP int serverport = 10010;                         Server port number try{//Establish connection DS = Newdatagramsocket (); Initialize send data Date d = new Date (); Current time String content =d.tostring ();                         Convert to string byte[] data =content.getbytes ();                         Initialize the Send Package object inetaddress address =inetaddress.getbyname (ServerHost); SENDDP = Newdatagrampacket (Data,data.lengTh,address,serverport);                                                                                             Send Ds.send (SENDDP);                         Initialize receive data byte[] b = newbyte[1024];                         RECEIVEDP = Newdatagrampacket (b,b.length);                         Receiving ds.receive (RECEIVEDP);                         Read the feedback content and output byte[] response =receivedp.getdata ();                         int Len =receivedp.getlength ();                         String s = newstring (Response,0,len);                 SYSTEM.OUT.PRINTLN ("Server-side feedback:" + s);                 }catch (Exception e) {e.printstacktrace ();                        }finally{try{//close connection ds.close (); }catch (Exceptione) {}}}}</span></span>

Socket Programming

Based on TCP protocol, a stable connection point-to-point communication is established.

Real-time, fast, high security, occupy system resources, low efficiency;

"Request-Response" mode

Client: In network communication, the first program to initiate the communication is called the client program.

Server: The program that waits for a connection in the first communication is called a server-side program.

Socket: Sends a TCP message.

ServerSocket: Creating a server

Sockets are an inter-process data exchange mechanism. These processes can be on the same machine or on different machines that are connected over the network. In other words, the socket acts as a communication endpoint. A single socket is an endpoint, while a pair of sockets forms a two-way communication channel that enables non-associative processes to exchange data locally or over the network. Once a socket connection is established, the data can be sent bidirectional or in one Direction on the same or different systems until one of the endpoints closes the connection.

URL (Uniform resourcelocator): Uniform Resource Locator, consisting of 4 parts: protocol, host domain name, resource file name, and port number where the resource resides.

A URL is a pointer to the Internet resource, which can be a simple file or directory, or a reference to a more complex object, such as a query to a database or search engine.

" Extended "

Uniform Resource Identifier, Uniform Resource Identifier;

Url:uniform Resourcelocator, Uniform Resource Locator;

Urn:uniform resourcename, Uniform Resource name.

Where Url,urn is a subset of URIs.

The basic form of an address on the web is a URI, which represents the Uniform Resource identifier. There are two kinds of forms:

URL: The most common form of URIs today is ubiquitous URLs or uniform resource locators.

An updated form of Urn:url, the Uniform Resource name (URN, Uniform Resource name) is not dependent on the location, and it is possible to reduce the number of failed connections. But its popularity will take time, because it requires more sophisticated software support.

A URI is a simple string that identifies a resource in a uniform (standardized) way.

Demo Chat room: Enable chat between the simple console and the server.

<span style= "FONT-SIZE:18PX;" ><span style= "FONT-SIZE:18PX;" >package com.bjsxt.wsl.s4;//helloserver class Import Java.io.ioexception;import Java.net.serversocket;import Java.net.socket;import Java.util.arraylist;import java.util.List; public class HelloServer {//declares a container that holds all currently accessed clients privatestatic list<serverthread> threadlist = NE          W arraylist<> (); publicstatic void Main (string[] args) throws IOException {System.out.println ("---------------I am a server-----                   -----------");                   Create a server Serversocketserversocket = new ServerSocket (18888);  Let the server wait for the Accept//server to start listening port waiting for client request while (true) {Socketsocket                            = Serversocket.accept ();                            Starts a thread that specifically obtains the client's information and returns the information to Serverthreadserverthread = new Serverthread (socket, threadlist);                Joins the current client to the list            Threadlist.add (Serverthread);                   Newthread (Serverthread). Start (); }}}===============================================================================package com.bjsxt.wsl.s4;// Hellosocket class import Java.io.ioexception;import java.net.socket;import java.net.UnknownHostException;                   public class Hellosocket {publicstatic void main (string[] args) throws Unknownhostexception, IOException {                   System.out.println ("-------------I am a client--------------");                   Create a client Socketsocket = new Socket ("192.168.1.164", 18888);                   Opening a thread is specifically responsible for receiving user input from the console and writing it out to the server Newthread (new Socketsendthread socket). Start ();         Opening a thread is specifically responsible for receiving the server's return information and printing newthread (new Socketacceptthread (socket)). Start (); }}===============================================================================package com.bjsxt.wsl.s4;// Serverthread class Import Java.io. Datainputstream;import Java.io.dataoutputstream;import Java.io.ioexception;import Java.net.Socket;import Java.util.List;         public class Serverthread Implementsrunnable {//package stream information Privatedatainputstream datainputstream = null;         Privatedataoutputstream dataoutputstream = null;         Privatesocket socket = NULL;          Privatelist<serverthread> threadlist = null;  /** * constructor * @param socket */publicserverthread (socket socket, list<serverthread>                            threadlist) {try{this.socket= socket;                            This.threadlist= threadlist;                            datainputstream= New DataInputStream (Socket.getinputstream ());                   dataoutputstream= New DataOutputStream (Socket.getoutputstream ());                   }catch (IOException e) {e.printstacktrace (); }} @Override PuBlicvoid Run () {while (true) {//Receive client information sent to the server SE                   Ndallsocket (Accept ()); }}/** * Send message to other people * @param msg */publicvoid Sendallsocket (String ms g) {//Traverse for (Serverthread serverthread:threadlist) {ser                   Verthread.send (msg);                   }}/** * will be careful to return to customer interrupt * @param msg */publicvoid Send (String msg) {                            try{Dataoutputstream.writeutf (socket.getinetaddress () + "said:" + msg);                   Dataoutputstream.flush ();                   }catch (IOException e) {e.printstacktrace ();                   }}/** * Start accepting messages from clients * @return */privatestring Accept () {              Stringmsg = "";     try{msg= Datainputstream.readutf ();                   }catch (IOException e) {e.printstacktrace ();         } returnmsg; }} ===============================================================================package com.bjsxt.wsl.s4;// Socketacceptthread class import Java.io.datainputstream;import java.io.ioexception;import java.net.Socket;  public class Socketacceptthread implements runnable{//encapsulates stream objects to prevent duplicates from being created Privatedatainputstream DataInputStream          = NULL;                            Publicsocketacceptthread (socket socket) {try{//create an input stream to get the return information of the server                   datainputstream= New DataInputStream (Socket.getinputstream ());                   }catch (IOException e) {e.printstacktrace (); }} @Override publicvoid run () {while (true) {AccepT ();                            }}/** * Receive and print the return information of the server */Privatevoid Accept () {try{                            Stringmsg = Datainputstream.readutf ();                   SYSTEM.OUT.PRINTLN (msg);                   }catch (IOException e) {e.printstacktrace (); }}}===============================================================================package Com.bjsxt.wsl.s4;//So Cketsendthread class import Java.io.bufferedreader;import Java.io.dataoutputstream;import java.io.ioexception;import Java.io.inputstreamreader;import Java.net.Socket; public class Socketsendthread Implementsrunnable {//encapsulates a reusable class privatebufferedreader BufferedReader = Nu         ll          Privatedataoutputstream dataoutputstream = null;                   /** * constructor * @param socket */publicsocketsendthread (socket socket) { try{//Accept UserInput from the console bufferedreader= new BufferedReader (New InputStreamReader (system.in));                   Send the client's information to the server dataoutputstream= new DataOutputStream (Socket.getoutputstream ());                   }catch (IOException e) {e.printstacktrace ();                            }} @Override publicvoid run () {///repeatedly listens for the client's input while (true) {                   Send ();                    }}/** * The first piece gets the input data of the user console * The second piece sends the input data to the server */privatevoid send () { try{//define input information stringmsg = Bufferedreader.readline                            (); Determines whether the information entered by the user is valid if (msg! = NULL &&! ").                            Equals (msg)) {Dataoutputstream.writeutf (msg);                 }  }catch (IOException e) {e.printstacktrace (); }}}</span></span>

Business Ideas

Network programming is as a programmer must be the body of the plan, the idea is very important, but also hope that we can quickly enter the world of network programming.

[Java Primary]:22. Network programming