Email blacklist whitelist gray list

 

Spam has become a serious problem. In order to win the war against spam, we use 18 weapons. The Blacklist, whitelist, and gray list should be the most basic tools in this battle against spam, this article will elaborate on how enterprises can effectively use them.

As postage costs keep rising, the increasing complexity of email is an inevitable trend. Before spam flood, the Internet was very calm, and the mail system connected based on the simple Email Transfer Protocol (SMTP) worked well. The mail filter was only used by specialized email service providers. However, email filters have become a necessity for many departments.

So what kind of filter should I choose? If your company receives a very large number of emails each day, black lists and whitelists may not filter out spam in most cases, and they can only solve their urgent needs. Subscribing to services such as Postini can alleviate this problem from the perspective of mail receipt, but this is only half the battle against anti-spam.

Free Domain Name Server blacklist (blacklist)-such as spamhaus.org, spamcop.net and other websites have this service, providing an interactive service. Based on this service, through simple DNS queries, the server receiving the email can compare the IP address of the email sending server with a known list of spam servers. If the IP address is in this list, this email will be rejected.

Many organizations also rely on white-list, which is a simple list of domains, IP addresses, and SMTP forwarding IP addresses that can receive emails from them. In most networks, this is a domain of a partner closely related to the company and a list of complementary IP addresses, or a list of domains that should be valid when captured by the spam filter.

In addition, the whitelist-based protection method is greylist ). The gray list is between the blacklist and the white list. It uses interpreted background programs and SMTP status markers to dynamically create the Black List and white list.

All three methods have their own location in the Anti-Spam warfare of modern enterprises, but they must be carefully planned, especially when using blacklists, so as to avoid injury to the innocent.

Blacklist first

Although many DNS blacklists are in use, their use remains controversial. If this blacklist is too large, it will make the email server unable to work at all. Fortunately, this situation has not yet occurred, in addition, the spam server listed in the DNS blacklist rarely finds "misjudgment.

Normal email servers may be blacklisted. There are many reasons for this situation: reporting the IP address of the spammers directly may not only cause this IP address, even the entire network segment of this IP address is blacklisted by DNS. Users of shared hosts can easily become victims. Because they use the same IP address, if a user violates regulations, all websites using this IP address will be affected; in another case, the end user of the ISP may mark the emails in the valid Email sending list as spam instead of canceling the subscription service. This server may be blacklisted, at least the ISP will be blacklisted.

Different service providers provide different lists with different focuses and scopes. The largest sorbs.net, spamhaus.org, and spamcop.net websites use a general spam guide to determine the status of a server. The Rfc-igno-rant.org goes further by blacklisting email servers that violate RFC 821 and 2821 (RFC 821 and 2821 are the main specification for SMTP communication ). Unfortunately, a considerable number of legitimate email servers violate these rules due to poor design or incorrect implementation. Users who use these email servers may be blacklisted by the rfc-ignorant.org, even if they are not spammers. Although these websites should use compliant servers, their blacklisted DNS may impede legal communication with others.

However, it is undeniable that the most popular DNS blacklist in the past few years has been greatly improved, providing users with more accurate results than before. In fact, the free blacklists such as spamhaus.org and sorbs.net not only list the network segments of common spam servers, it also lists the dynamic IP addresses of spammers connected through home broadband and botnets controlled by hackers to send spam.

How popular are these blacklists? According to Steve Linford, who works at spamhaus.org, The spamhaus network receives 80 thousand to 0.1 million search requests per second. This does not include member organizations of large organizations that do not use public servers. These large organizations regularly obtain DNS blacklists from public servers as planned, but put them in their networks for use by lower-level members, this greatly reduces the number of user requests to the public server.

Blacklist misjudgment

But what is the false positive rate? "We didn't use the DNS blacklist until last night due to fear of misjudgment," he said. However, in the past few months, the number of spams we have received has increased rapidly. As a last resort, I decided to use the njabl.org blacklist in our email filter. We have blocked more than 3100 connections in the past 15 hours ."

If the DNS blacklist is popular, misjudgment will always exist objectively. However, because the advantage of using the blacklist is far greater than the disadvantage, this worry is nothing compared with the increasing problem of spam.

After a server is blacklisted, the website administrator usually does not know until a large number of rejected emails are returned to the user. In most cases, the returned information includes the reason why the email is blocked, who blocked the email, and so on. A warning usually includes a URL to instruct administrators on how to apply to remove their email servers from the blacklist. It is estimated that 0.5 million servers in spamhaus.org are blacklisted every day.

Each DNS blacklist has its own unique method for collecting and maintaining its database. Many uses the honey Network Technology (Honeynet) to automatically classify attacks from botnets. If botnets are found, they will add source IP addresses to the database. The dead-end SMTP technology is also frequently used, and they do not have a real mailbox, however, emails sent to non-existent users are collected to identify spam websites and systems.

Despite the fact that open relay is no longer a threat on the Internet today, it still exists. Several organizations that provide DNS blacklists will actively search for open forwarding. Once discovered, they will be blacklisted.

Not long ago, in many commercial SMTP servers sold, open Forwarding is still set by default. But it is no longer used today. However, John Gilmore, one of Sun's first employees, founder of the Cygnus solution for eff and father of the Usenet alt news group, also insisted on retaining the Limited Open forwarding function. For him, this is a question of freedom of speech. But for us, it is not a good practice, it will make the email basically invalid.

Gray list becomes popular

The gray list can cleverly intercept most junk emails. Its main function is based on the SMTP error code, which means that the sender needs to wait a few minutes before sending the email again.

This code is usually sent when the email server receives too many requests and cannot process them. The gray list is based on the fact that most spam servers and botnets send emails only once, and ignore the requests that require them to send again after a certain interval. Because for them, resending each mail will greatly reduce their total business volume.

All emails that are initially rejected by the email server and requested to "resend later" will go to the gray list filter. If the remote server resends the email in about 10 minutes, it will pass the email without any obstacles, and the email with the same header will pass smoothly in the future.

Recently, gray lists are becoming increasingly popular. This method can greatly reduce the number of spam mails, but it requires the server to send emails again and delay the receiving of emails. However, this delay is necessary to identify whether it is spam.

Even so, adding one or more DNS blacklists to the gray list, and adding junk and virus filters can provide us with a relatively clean email system. Today, they have become an essential standard for SMTP servers to prevent spam and viruses. Although there are still opportunities to lose emails, it is not a fatal problem.

Final Solution of Spam

To truly solve Spam's troubles, we still need some truly groundbreaking technologies. SPF (sender policy framework) is a technology that can really deal with spam ). SPF is essentially a reverse validation of each received email.

Just as each Internet mail server requires an mx dns record for receiving mail, SPF requires each server to have a record for sending MX. That is, a DNS record in a domain can be used to verify that a server is responsible for sending an email. If an email server using SPF finds that the email sending server has no records in the domain's DNS, the email sent by the server will be returned or marked as suspected spam. For example, the server receives an email claiming to be from aol.com

The email server cannot be found in. com, so this email may be forged.

This solution also has advantages and disadvantages. For example, if MTA (mail transfer agent, mail Transmission proxy) fails to forward the email, the server is required to resend the email instead of resending the email when the SPF filter is used. These technologies are still in the development process.

Another option is to use the X.509 Certificate to protect SMTP. This method requires that each valid SMTP server on the Internet has a corresponding ID card. Only servers with valid certificates can send emails to another server. This solution requires that most of the currently running mail servers have certificates; otherwise, they cannot be sent or listed as pending.

Although SPF has become increasingly popular recently, a well-developed solution is unlikely to appear soon. Unless several major open-source and commercial MTA product providers start to cooperate on the same standard, the blacklist-based email receiving system will still be the main method. (Translated from inforworld magazine)

Link: Magic dashboard

Although the primary DNS blacklist websites provide their services to most users free of charge, these services require costs. As DNS blacklists become increasingly popular and more effective, they pose a great threat to the interests of large-scale spam recipients and their customers. Therefore, the DNS blacklist provider finds that they have been involved in a battle with spam senders, but it is not how to deal with spam.

A sorb.net employee said: "This is indeed a war and is being upgraded. We are actively trying to find and block spam makers, and they are trying their best to destroy us ." "For example, we will scan open relay generated by malware, some malware programmers confuse our scanning programs by replying invalid information, leading to repeated scanning. This reduces the validity of the scan, so we have to modify the scan program to avoid this problem ."

There are also stories about espionage and dual espionage in this war. The sorb.net staff also recalled one thing. Someone once sent an anonymous letter to sorb.net saying that if a specific 24-byte data was sent to the TCP port, some Windows malware on this computer will be automatically uninstalled. After receiving the message, sorb.net modified the scan program and added the sequence. Then, it was found that the virus on thousands of infected computers had been cleared.

Although DNS blacklists use various methods to compile their databases, Spammers can still identify and try to escape. For example, spam manufacturers will specifically develop malware to block connections from well-known DNS blacklists to avoid being scanned. Other technologies include the "anti-Black List" of the DNS blacklist, that is, the spam maker sorts out the list of server addresses frequently used by the DNS blacklist service provider to scan, in this way, they can take targeted measures.

In addition to cat and mouse games, the DNS blacklist and spam senders also use DDoS to attack large DNS blacklists. Not long ago, spamhaus.org suffered a lot and was forced to adopt anti-DDoS to maintain its services.

The current situation is to avoid and attack, escape and detour. One party tries every means to win over the other. If Windows XP SP2 and the coming Vista are more secure, the spam maker's tricks may not be that easy to achieve. However, this is simply "if ". At present, it seems that the struggle between the two sides will continue.