Deep Web request process (note)

Deep Web request process B/. Benefits of S-Architecture:

b/S architecture brings two benefits:

L The customer service side uses the unified browser, because the browser has the unification, effectively blocked the different service providers to provide the user to use the services the difference. 、

The service-side is based on unified HTTP, simplifying the development model for service providers and saving development costs, and service developers only need to focus on the application logic that provides the service.

b/S Network architecture overview

b/S network architecture from the front end to the back end are based on the Unified Application layer protocol HTTP to interact with the data, and traditional C/s architecture with the use of long-connected interactive mode, HTTP with a stateless short-link communication mode. Typically, a single request completes a data interaction, usually a business logic, and then the communication connection is disconnected. This is used in order to be able to serve more users at the same time.

As shown in: The user in the browser input URL, first request DNS to resolve the domain name to the corresponding IP address, and then based on this IP address in the Internet to find the corresponding server, send a GET request, the server returns the default data resources to the user access. If you have more than one server, you also need a load balancer device to evenly distribute all users ' requests.

How to initiate a request

The process of initiating an HTTP request is to establish a socket communication process. Before the browser establishes the socket connection, it must resolve the IP address according to the domain name of the URL written in the address bar, and then establish a socket connection to the remote server based on the IP address and the default port 80, and then the browser is assembled into a get-type HTTP request header based on this URL. Sent to the target server, the server sends back data, disconnects.

With Curl+url, you can simply initiate an HTTP request under Linux. View HTTP header information, plus the-i option

HTTP parsing

Common HTTP request headers:

Accept-charset: Used to specify the character set accepted by the client

Accept-encoding: Used to specify acceptable content encoding

Accept-language: Used to specify a natural language

Host: Intent host and port number with the specified requested resource

User-agent: Client will tell the server about his operating system, browser and other properties

Connection: Whether the current connection is maintained

Common HTTP corresponding headers:

Server: Using the name

Content-type: The type of media used to indicate the body of the entity sent to the recipient

Content-encoding: Corresponds to the request header accept-encoding to tell the browser what compression encoding is used on the server

Content-language: Describes the natural language used by the resource.

Content-length: Indicates the length of the entity body, expressed as a decimal number stored in byte mode.

Keep-alive: Time to save the connection

Common HTTP Status:

200: Client Request succeeded

302: Temporary jump, the address of the jump is specified by location

400: Client request has syntax error and cannot be recognized by server

403: The server receives the request but refuses to provide the service

404: The requested resource does not exist

500: Unexpected error occurred on server

Browser caching mechanism:

Browser caching is a more complex but important mechanism. Pressing CTRL+F5 to refresh the page will re-request the server, even if it is a pull server, it is possible for the front end of the server to deploy a cache server, in order to allow users to see the latest data, must be controlled via HTTP. Add Pragma:no-cache and Cache-control:no-cache to the request header.

The HTTP head field is used to specify the instructions that all caching mechanisms must obey in the entire request response chain.
Optional values for the HTTP head field:

Public: All content is cached in the response header settings.

Private: The content is cached only in the private cache, in the response header settings.

No-cache: All content is not cached

No-store: All content is not cached in the cache or intent temporary files

Must-revalidation/proxy-revalidation: If the cache content fails, the request must be sent to the server for re-authentication

MAX-AGE=XX: Cache content expires in XXX seconds, this is only available in HTTP1.1.

Expires: The usual format is: expries:sat,25 12:22:17 GMT followed by a date and time, the cached content will expire if it exceeds this time value. That is, the browser checks this field of the page before making a request.

Last-modified/etag:

The Last-modified field is typically used for resource last modification times on a server, Resources can be static (static content automatically plus last-modified) or dynamic content (the servlet provides a getlastmodified method to check whether a dynamic content has been updated), and this last modification time determines whether the currently requested resource is up to date.

The browser requests the child again. The request header adds a if-modified-since: field that asks if the currently cached page is up to date. If it is up to date, the server returns 304.

ETag This field lets the server assign a unique number to each page, distinguishing whether the current page is up to date. If more than one server, it is more difficult to handle.

DNS Domain name resolution

Linux and Windows can use Nslookup to query the results of domain name resolution, and Linux can also use dig naming to query the DNS resolution process. Plus trace to track the domain name resolution process

After DNS parsing, the parsing results are cached, with the main two cache results, one local domain name server and one user local machine.  This two cache is the TTL value and the native cache size control, the local machine can use the IPCONFIG/FLUSHDNS command to flush the cache. Linux is cleared by/ETC/INIT.D/NSCD Restartlai.

The JVM in Java also caches DNS parsing results. This cache is in the InetAddress class, there are two caching strategies: one is to parse the result cache correctly, and the other is the failed parse result cache. This two cache time is configured by the Lib\security\java.security file, Networkaddress.cache.ttl and Networkaddress.cache.negative.ttl Default values-1 (never expire) and 10 (cache 10 seconds)

Need to use the InetAddress class to resolve the domain name, must be a singleton mode, or there will be serious performance problems. Every time you create a inetaddress= instance, you perform a full domain name resolution every time.

Domain name resolution records are divided into a record, MX record, CNAME record, NS record, TXT record

A record: Used to specify the corresponding IP address. A record can resolve multiple domain names to an IP address, but cannot resolve multiple IP addresses with a single domain name.

MX: Record: is a domain name under the mail server to its own mailbox server.

CNAME: Alias parsing. The so-called alias resolution is to set one or more individual names for a domain name.

NS RECORD: Specifies the DNS resolution server for a domain name, which is the DNS server that has the specified IP address to parse.

TXT record: Set a description for a host name or domain name.

CDN Working mechanism

CDN is the Content Division network. is an advanced traffic distribution network, which aims to improve the corresponding speed of users to visit the site, different from the mirror, more intelligent than mirroring. At present, the static data is the main, the user requests from the master server to the dynamic content, and then downloads the static data from the CDN, thus speeding up the download speed of the webpage data content.

Load balancing: balancing work tasks. Allocated to more than one unit of action execution. It can improve the response speed and efficiency of the server, avoid the single point failure of software or hardware, solve the network congestion problem, realize the geo-independent line, and provide the users with consistent access quality.

Deep Web request process (note)