Establishing an email system with large capacity web interface

Web in recent years, web-based free email system is very popular, at present, several famous free email websites have basically become the majority of people's choice, the establishment of a simple free email service site is no longer like the previous warm welcome, But the email service that provides the Web interface has become one of the basic services that a business site provides for its registered members.

An email system can be divided into server and client, the Web interface of the email system is the email customers on the Web server side, so the email system needs to implement a Web interface email customers. However, because this email system requires a large number of users, so there are specific requirements for the email server.

1. Operating systems and user databases, which require stability and performance for web and email services, generally use UNIX as the server's operating system, for example, Hotmail uses FreeBSD and Solaris, and domestic 163 sites are also BSD series. However, Unix's standard email system is not suitable for this bulk service. Some UNIX systems, such as the current version of Linux, have a user ID of only 16 digits, so the number of users can only be 64K, even if the UNIX system itself supports a 32-bit user ID, the number of users supported by a single server does not exceed 100,000, given the performance factor.

In order to have the scalability to support more users, the general use of multiple servers at the same time to provide services, although still can use standard UNIX users as email users, but in view of security, performance and manageability, generally use non-UNIX system users as email users. While the user data is usually used to support the network Access database form, commonly used in the LDAP, standard database, and the email system to achieve their own user database. Among them, LDAP is the standard to provide directory service, so it should be the best choice, its common open source code is implemented as OPENLDAP, and standard database is convenient and extensible, which is the most commonly used in the int ernet for MySQL, in addition, there are other ways to implement.

2. Because of the large number of users, how to save the user's mail is a very important issue. Traditional UNIX uses a single directory to hold all users ' messages, which greatly reduces the performance of the file system when the number of users is large. Only a multi-level directory, a limited number of files per directory, can reduce the system consumption when opening files, or no longer use simple files to save messages, but in a certain package form. Completely take the database form to save the mail, because the user mail operation is many files operation, and the size change is big, therefore can cause the performance and the storage space bigger waste.

Because of the large number of users, and also require access to multiple servers at the same time, you must use a large storage server or server cluster to save, through the Fibre Channel or network File system NFS to share storage space, so that each user's mail storage path for each server is consistent. Fibre Channel is a very expensive solution, and more commonly used is the use of NFS, dedicated Server for NFS, such as NetApp, or the use of a RAID-capable PC UNIX server.

When using NFS to share storage space, a very important issue must be noted. Due to the lack of file locking mechanisms for NFS, the traditional user message storage format is mailbox, because all messages are kept in the same file, so mail operations must be locked to ensure no access violation, making it unsuitable for NFS storage. To solve this problem, QMail put forward the Maildir storage mode, each message as a separate file saved in the user's personal mailing directory, to avoid the lock. Therefore, the common free mail server, generally uses the Maildir way to save the user's mail.

If you do not intend to use a shared file system to save a user's mail, and you intend to have each server access only the user's Mail on its own hard disk storage space, the email server and the client will need to be customized so that they can find the real server that the user belongs to through the username. Hand over the access task to this server for completion. The disadvantage of this method, in addition to the need for larger changes in the system structure, but also because the server is divided by users, is not conducive to sharing load. The advantage is that it does not access other servers over the network, so you can use any of the message storage formats, including the use of powerful Cyrus systems to save messages and provide services.

3. mail server Software standard email software, such as SendMail, although also provides some methods including aliases, to support non-UNIX system users, but these capabilities for the implementation of this email system is not enough. In order to support these email users, you must use your own email server software. Generally because of the existing email software is quite mature, but also open source software, it is generally to change the original email software, such as SendMail, qmail, etc., so that it supports specific email users. Completely rewriting an email service software is not desirable from the perspective of maturity and stability.

Regardless of performance, security considerations, SendMail is not an ideal choice, and because the qmail itself to support the Maildir, it has become a commonly used email Software Foundation development platform. It is important to note, however, that QMail uses GPL licenses for protection, so any changes based on qmail must in principle expose the source code, which can be a hindrance to the development of the commercial system. Of course, you can change the relevant system library functions by not changing the qmail, or use the way to bypass the problem. Another optional basic email software is postfix, which has its own interface with LDAP and MySQL, and can act as part of the messaging system with little change.

4. Another important part of the Web Client Web Interface email system is the Web client, which functions like outlook in personal computers and is responsible for providing users with the ability to access their own mail. Because Web Access itself is connectionless, the security of the user must be ensured. Basically, security can be guaranteed by the session identification, the temporary directory, and the validation in the program that is established after the login.

Web clients must access the server in a uniform manner, through direct file access to the user's mail, or through standard protocols such as POP3 and IMAP. For systems that use network file systems to share user mail, the direct file access method is most straightforward and convenient, and does not require additional consumption. And through POP3, IMAP protocol to access the server, its direct benefit is the Web client and email server separation, improve the system security.

Currently, there are quite mature open source Web client software, where IMP is implemented using PHP to access the server's Web mail client software through an IMAP protocol. And wing is another Web client software that is implemented using Perl. These open source software are pretty good, but integrating these software into your system will require some changes. In addition, you should follow its licensing requirements to expose the changed code to the outside world.

5. To achieve load balancing because of the need to provide a large number of users to access, so a single server does not meet this need, but must use a multiple-server approach. In addition to functional segmentation, such as the separation of Web servers, email servers, and file servers, several resource-intensive services need to be load balanced using multiple servers. Although some of the current commercial manufacturers also put forward some server cluster scheme, but the common simple and effective method is DNS loop resolution, Web server relocation and NAT load balancing.

DNS looping resolution is to assign multiple IP addresses to the same name, which is used on quite large sites such as Yahoo, and it actually works pretty well. The Web server relocation is the Web server randomly generated on different servers in the real page URL, so that different browsers load different servers on the page, use it can only achieve a Web client load balance. While NAT load balancing utilizes layer fourth switches to turn the same requests to different servers, and there are some software that can complete the NAT function in addition to expensive switches, I have changed the FreeBSD natd to support load balancing, It is also an option for users who have to reduce performance requirements because of the switch price issue.

6. Example analysis of the current domestic most popular web interface email system for NetEase Company's system, it is the use of qmail as the basic server software, and then change the system. It uses the NFS network file system as the user mail storage space, uses the Maildir as the Mail storage format, provides the multi-level directory to support the large number of users. Their web clients are implemented by themselves, providing services to users by accessing their mail directly. This implementation is a very popular and mature way to do not consider the small problems of its software, and most free mail service systems use this pattern.

Another way is to make the most of the open source software available, a viable solution is to use Postfix, OpenLDAP, Cyrus and Imp to implement a large-capacity email system, where the primary mail server uses the Postfix query LDAP server to determine the user's real e-mail address , and then forwards to the real mail host, which is confirmed by the LDAP query, puts the message into the Cyrus server, and IMP accesses the user's mail using IMAP via login Cyrus. When the user is increasing and a Cyrus server is not enough, the newly added user can be placed on the newly added server, just set the appropriate properties on the LDAP server. In this way, because the user is strictly by the server division caused by management and other difficulties, the structure itself is more complex. However, if the number of users is not much, then there is no need to use multiple Cyrus servers and LDAP servers, the complexity is greatly reduced, more suitable for small and medium-sized sites.