SMTP protocol Analysis

mail server server documentation domain login command

Directory (?) [+]

SMTP Protocol Analysis     1th Chapter.  SMTP Overview 1.1. The location of SMTP in mail communication

SMTP, the Simple Mail delivery protocol, corresponds to the RFC document RFC821. As with most application layer protocols like HTTP, it works in C/S mode and is used to deliver mail on the Internet. The location of SMTP in the entire e-mail communication is shown in Figure 1.

Figure 1 Communication process for e-mail

As you can see, SMTP is used to deliver messages on the client computer to the server. The client here refers to the sender of a connection, and the server refers to the corresponding receiver. Before explaining the entire communication process of sending an email, explain the following terms first.  1.2. Several terms 1.2.1. Mail

A message is a format of messages that consists of envelopes, headers, and bodies.

The most important thing on the envelope is the address of the addressee. The mail server uses this address to send messages to the mail server where the recipient is located.

The header is some information added by the user agent or mail server. Includes fields such as received, Message-id, from, Data, Reply-to, X-phone, X-mailer, to, and subject.

The text is the content of the sending user's message to the receiving user. RFC 822 Specifies that the body is a NVT ASCII line of text.

For more detailed instructions, please refer to the RFC821 and RFC822 agreements. 1.2.2. User Agent

The user Agent UA is the interface between the user and the e-mail system, which in general is a program on our PC. Common user agents on Windows are Foxmail and Outlook Express.

The user agent provides a good user interface, which extracts the information that the user fills in its interface, generates a mail message that conforms to the SMTP standard, and then sends the message to the sending-side mail server using the SMTP protocol. 1.2.3. Mail server

A mail server is the core of an e-mail system that is used to send and receive messages. The mail server differs from the normal PC in that it works almost all day, so it can serve the user at any time, and it will be mentioned that this is one of the important reasons why a mail server is needed. Many ISPs offer free mail servers, such as 126 for smtp.126.com mail servers.

The SMTP protocol is also used by the mail server to forward mail to other mail servers. 1.3. Mail sending and receiving process

In general, the sending and receiving process for a message is as follows.

1) the sender in the user agent to edit the mail, including fill in the sender's mailbox, recipient mailbox and mail title, and so on.

2) The user agent extracts the information from the sender's editor and generates a message in accordance with the standard message format (RFC822).

3) The user agent sends the message to the sending-side mail server (the mail server corresponding to the sender's mailbox) with SMTP.

4) The sending mail server uses SMTP to send mail to the receiving End mail server (i.e. the mail server corresponding to the recipient's mailbox).

5) the recipient invokes the user agent. The user agent uses the POP3 protocol to retrieve messages from the receiving-side mail server.

6) the user agent resolves the received message in the appropriate form present in front of the receiving person.     2nd Chapter.  SMTP detailed 2.1. Communication Process

The process for a specific SMTP communication, such as communication between a sending-side mail server and a receiving-side server, is as follows.

1) The sending-side mail server (hereinafter referred to as the client) establishes a TCP connection to port 25th of the receiving-end mail server (hereinafter referred to as the server).

2) the client sends various commands to the server to request various services (such as authentication, designated sender, and recipient).

3) the server resolves the user's command, makes the corresponding action, and returns a response to the client.

4) 2) and 3), until all messages have been sent or the connection between the two has been interrupted unexpectedly.

As seen from this process, commands and responses are the focus of the SMTP protocol, which is highlighted below.  2.2. Command and Response 2.2.1. Format

SMTP commands are not many (14), and its general form is: command [Parameter] <CRLF>. Where command is the name of the ASCII form, parameter is the corresponding command parameter,<crlf> is the carriage return newline character (0DH, 0AH).

The response of SMTP is also not complex, its general form is: XXX readable illustration. XXX is a three-bit decimal number; Readable illustration is a readable explanatory note used to indicate whether the command was successful or not. XXX has the following rules: The expression beginning with 2 succeeds, the expression beginning with 4 and 5 fails, and the representation beginning with 3 is not completed (in progress). 2.2.2. An example

The format of commands and responses is syntax, and each command and response is semantic, and each command and response is synchronized in relation to time. The three elements of the protocol are described below through a simple SMTP communication process.

C:telnet smtp.126.com 25/* Telnet connection to 126 mail server */

s:220 126.com Anti-Spam GT for Coremail System (126com[071018])/* 220 is a response number, followed by welcome information, will vary depending on the server */

C:helo smtp.126.com/* HELO is used to fill in the return domain name (see RFC821 for details), but the command does not check the following parameters */

s:250 OK

C:mail from:bripengandre@126.com/* Sender Mailbox */

S:250 .... /* "..." representative omitted some readable information */

C:RCPT to:bripengandre@smail.hust.edu.cn/* Recipient mailbox */

S:250 .... /* "..." representative omitted some readable information */

C:data/* Request send data */

s:354 Enter Mail, End with "." To a line by itself

C:enjoy Protocol studing

C:.

s:250 Message Sent

C:quit/* Exit connection */

s:221 Bye

Analysis of the above process can be referred to the comments, here to add the following points.

1) the line starting with "C:" (excluding "C:") is the client's input, and the line that begins with "s:" (excluding "s:") is the output of the server.

2) The above command does not necessarily succeed at one time, the server returns an error response, and the client should enter subsequent commands (or repeat the failed command, or reset the session, or exit the session, etc.) in accordance with the timing specified in the protocol. 2.2.3. Common Commands

The SMTP command is case-insensitive, but the parameters are case-sensitive, please refer to RFC821 for detailed instructions. The common commands are as follows.

HELO <domain> <CRLF>. Identify the user to the server the sender can cheat and lie, but in general the server can detect.

MAIL from: <reverse-path> <CRLF>. <reverse-path> is the sender's address, which is used to initialize the message transfer, which is used to initialize all states and buffers.

RCPT to:<forward-path> <CRLF>. <forward-path> is used to flag the email recipient's address, which is often used in mail from, and can have multiple rcpt to.

DATA <CRLF>. Send the following data as data to <CRLF>.<CRLF> mark the end of the data.

REST <CRLF>. Resets the session and the current transfer is canceled.

NOOP <CRLF>. Requires the server to return an OK response, which is typically used for testing.

QUIT <CRLF>. Ends the session.

VRFY <string> <CRLF>. Verify that the specified mailbox is present and that the server mostly prohibits this command for security reasons.

EXPN <string> <CRLF>. Verify that the given list of mailboxes is present and that the server mostly prohibits this command for security reasons.

Help <CRLF>. What commands are supported by the query server. 2.2.4. Common Responses

The usual responses are as follows, and the numbers are translated from English. Please refer to RFC821 for more detailed instructions.

501 Parameter Format error

502 Command not implemented

503 Wrong sequence of commands

504 command parameters are not implemented

211 System status or system help response

214 Help information

Service Ready

221 Service shutdown

421 Service is not ready to close the transport channel

250 Required mail operations completed

251 The user is not local and forwards to the

450 The requested mail operation is not completed and the mailbox is not available

550 The requested mail operation is not completed and the mailbox is not available

451 abort the requested operation; Error during processing

551 users are not local, try

452 Insufficient system storage, required operation not performed

552 Excessive storage allocation, required operation not performed

553 The mailbox name is not available and the requested action is not performed

354 Start the Mail entry to "." End

554 operation failed Chapter 3rd.  Expansion of SMTP 3.1. Disadvantages of SMTP

As you can see from the 2.2.2 example, SMTP has at least the following drawbacks.

1) the command is too simple to provide authentication and other functions.

2) only 7-bit ASCII code is transmitted, not binary files.

For disadvantage 1), the standardization organization developed an expanded SMTP (i.e., ESMTP), and the corresponding RFC document is RFC1425. For the disadvantage of 2), the standardization organization on the premise of compatibility with SMTP, the transmission of non-7-bit ASCII code, the corresponding RFC document has two: the extension of the message header corresponds to RFC1522, the message body extension corresponding to the RFC1521 (that is, MIME). 3.2. ESMTP

ESMTP the most significant place is the addition of user authentication features. If the user wants to use the new command provided by ESMTP, the command sent should be EHLO instead of helo at the first interaction with the server. Let's look at an example first.

C:telnet smtp.126.com 25/* Telnet connection to 126 mail server */

s:220 126.com Anti-Spam GT for Coremail System (126com[071018])/* 220 is a response number, followed by welcome information, will vary depending on the server */

C:ehlo smtp.126.com/* In addition to Helo features, EHLO is primarily used for querying server-supported extensions */

S:250-mail

S:250-auth LOGIN PLAIN

S:250-auth=login PLAIN

S:250 8BITMIME/* The last response to a digital answer code is followed by a space, not '-' */

C:auth LOGIN/* Request authentication */

s:334 DXNLCM5HBWU6/* Server Response--Base64 encoded "Username" */

c:y29zdgfayw1hegl0lm5lda==/* Send a BASE64 encoded username */

s:334 UGFZC3DVCMQ6/* BASE64 encoded "Password:" */

c:mtk4mjixna==/* The BASE64 encoded password sent by the client */

s:235 Auth Successfully/* Certified Successful */

C:mail from:bripengandre@126.com/* Sender Mailbox */

S:250 .... /* "..." representative omitted some readable information */

C:RCPT to:bripengandre@smail.hust.edu.cn/* Recipient mailbox */

S:250 .... /* "..." representative omitted some readable information */

C:data/* Request send data */

s:354 Enter Mail, End with "." To a line by itself

C:enjoy Protocol studing

C:.

s:250 Message Sent

C:quit/* Exit connection */

s:221 Bye

There are several explanations for this example.

1) just a schematic process, and then enter the user name, password to use the Base64 code, which requires special calculation, so in the Telnet terminal simulation is more troublesome.

2) There are many kinds of certification process, there are clear-based authentication, but also based on MD5 encryption authentication, here is only a schematic process.

3) EHLO for the specific server, the response will be different, the keyword "8BITMIME" to indicate whether the server supports the body to transfer 8-bit ASCII code, and "X" is the beginning of the keyword refers to the server custom expansion (not included in the RFC standard)

For a more detailed explanation, please refer to RFC1425. 3.3. Extension of the mail header

The header supports the transfer of non-7-bit ASCII code by two encodings. It first uses a coded word in the following format to indicate how it is encoded.

=?charset?encoding?encoded-text?text

CharSet is a character set specification. Valid values are two strings us-ascii and Iso-8859-x, where x is a single number, for example, the number in iso-8859-1 is "1".

Encoding is a single character used to specify the encoding method that supports two values.

Q represents quoted-printable (printable) encoding. Any word to be sent Fu Jochi 8th bit 1 is sent as 3 characters: The 1th is the character is "=", and the following two characters correspond to the hexadecimal representation of the character. For example, for binary code 11111111, its corresponding hexadecimal representation is "FF", so the corresponding encoding bit "=FF". In order to be able to transfer "=", "=" is encoded in the same way as the 8th bit 1 character, because the binary code is 00111101, so the corresponding encoding is "=3d". It can be seen that this coding method costs up to 200%, so it is only suitable for transferring text containing only a small number of non-7-bit ASCII codes.

B stands for base64 encoding. It is encoded by dividing the binary code into a 24bit long unit and dividing the unit into 4 6-bit groups. Each group is converted to ASCII by the method shown in Figure 2.

Figure 2 Base64 Mapping table

It can be seen that this mapping method is as follows: 0-25 maps into a-z,26-51 in turn mapped into a-z,52-61 in turn mapped to the number 0-9, and then 62 mapped to +,63 map into/.

For binary code 01001001 00110001 01111001, first divide it into 4 6 bit groups, which is 010010 0100011 000101 111001. Then, as shown in Figure 2, the mapping table can be Base64 encoded as: STF5. As you can see, the cost of this encoding is 25%, which is more suitable for transmitting binary files with a large number of non-7-bit ASCII codes relative to quoted-printable encoding. 3.4. Extension of the text

The main body of the expansion is to enable the body not only to transmit NVT ASCII characters, and can transmit any character, the corresponding document is RFC1511 (that is, MIME).

MIME is all called "Multiple Internet Mail Extensions" and the exact Chinese name is "Multipurpose Internet Mail Extension". It makes it a widely used e-mail specification that can transmit multimedia by adding some message header fields, message content formats, and transfer encodings.

Refer to another article, "MIME Protocol Analysis" and RFC1511 for more detailed instructions.     4th chapter.  Common question 4.1. Why SMTP server is required

General PC resources are not enough, processing power is not enough, it is impossible to connect on the Internet to send and receive mail. So using an SMTP server allows multiple users to share the server, effectively reducing costs. 4.2. SMTP and message format relationships

As mentioned earlier, SMTP is the protocol used by the client to send mail to the server, the core of which is the command and response described in 2.2, and the format of the parameters in its commands and responses, which are dependent on other standards. For example, data after the parameters, you should follow the message format standard RFC822.

The SMTP and message format relationships can be illustrated in such an example. A and B correspondence, a through the post office letter to B, the Post office to transfer mail can be seen as the use of SMTP protocol, as for the format of the letter will be due to regional habits and other different (the Chinese letter format and the American letter format is different), this letter format can be regarded as the standard mail format.

It should be recognized that the agreement can not be viewed in isolation, there is often a coupling relationship between the agreements, but for the sake of analysis, we can only grasp the main contradiction when we describe an agreement in detail--mainly on a single agreement. 4.3. What protocol does the browser use to send mail?

Browsers such as IE, Maxthon can be logged in the user's mailbox, to send and receive mail, this is how to achieve. For example, bripengandre@126.com can send and receive mail by logging in to www.126.com.

The process is as follows: Bripengandre@126.com the corresponding information (such as the sender's email, recipient's mailbox, etc.) on the e-mail page provided by www.126.com, which is submitted to the 126 server via the HTTP protocol. ; 126 the server assembles a mail-compliant message based on this information (just like a user agent), and then smtp.126.com sends the message to the receiving mail server through the SMTP protocol.

As can be seen, the browser sends the message is only the user agent function directly to the mail server to do, as to the mail server between sending mail or the SMTP protocol used. We look at the problem, if necessary, to see the essence through the phenomenon properly. 4.4. How to verify the SMTP communication process with an experiment

1) The Packet Analysis protocol can be captured by ethereal and other protocol analysis software.

2) You can use socket programming to implement the SMTP communication process.

3) You can use the user agent to view the original encoding of a message. For example, in Foxmail, you can select "Raw information" from the right-click menu of the mailing list to view it.     5th Chapter. Analysis Solutions

Id	Protocol	Captured contents
User name	Password	Sender	Receiver	Subject	Contents	Attachments
4	Smtp	√	√	√	√	√	√

Table 1 Protocol Analysis requirements

Table 1 shows the requirements for protocol analysis. It is easy to see that it is easier to get the individual fields. We can crawl client-server interaction information, and then extract the fields we want based on each command word or response word. For example, to get the user name, we only need to detect that the server-side requires the client to send the username this time, and then extract the client's sending information after that. It is necessary to note that while the information that the client interacts with the server may be encoded or encrypted, we can still decode or decrypt it to obtain the required information.     6th chapter. References

[1] RFC document: RFC821 corresponds to the SMTP protocol, RFC822 corresponding to the message standard, RFC1425 corresponding to the esmtp,rfc1522 corresponding header extension, RFC1521 corresponding message body extension, RFC1939 corresponding POP3 protocol.

[2] http://www.faqs.org/rfcs/, with full English RFC documentation

[3] http://www.cnpaf.net/, there are many useful protocol analysis documents, there are Chinese RFC documents, but the quality is not particularly high

[4] Stevens, W.R, TCP/IP illustrated, Vol1. Addision-wesley, mechanical industry Press, 2002