FreeBSD.org's intrusion

Before returning home, the FreeBSD Security Team was preparing an important security bulletin, but this time it was not because of the FreeBSD Code itself, but because some of the FreeBSD project clusters were intruded. This announcement was officially published on August 1, November 17.

After analysis, this intrusion can be traced back to an intrusion targeting a company in around September this year. The attacker obtained the root account of a machine of the company and obtained the corresponding master. passwd file. Based on post-event log analysis, attackers should use offline cracking to obtain the plaintext of all users on the machine (the default hash mode of the old FreeBSD system is salted MD5 ), using this machine as the stepping stone, we scanned a series of other systems and found a committer's unprotected ssh private key. This private key allows attackers to gain permissions on another server (including root permissions on one server ), on that server, we further obtained another unprotected ssh private key from the committer.

The ssh private key of the committer has the root permission of the Pointyhat cluster used by FreeBSD to compile the pre-compiled package. This cluster is isolated from the FreeBSD.org cluster to some extent, however, at the initial creation, Filer in the/home directory of all users in the FreeBSD.org cluster was mounted in read-only mode for convenience. Because both clusters enable the kernel audit mechanism (audit), the Cluster Administrator finds that some users access the/home directory as root, it is likely that some private key files are copied.

At this point, we believe that the entire cluster has encountered major security risks. We should immediately stop the operation of risky servers and audit all files based on backups. In addition to powering off the power of most servers, scanning and disabling all logon permissions for public keys that have private keys on the FreeBSD.org Cluster, replace the operating systems running on some systems with audited-CURRENT.

Due to the time relationship, it is impossible to re-audit each Binary Package. Therefore, it is easier to delete all files, restore the files from the backup, and re-compile other Binary packages. In order to restore the service as soon as possible, this discussion also made a plan to prepare for the early elimination of CVSup.

Changes visible to the public include:

• In the future, all CVS exports will be performed as "svnexp. This script extracts the code from svn and commit it to CVS.
• All FreeBSD.org services that currently rely on CVS will be transitioned to using svn as the source. In the future, we will no longer provide CVS and CVSup services.
• At present, the pkgng website has not been restored, and the specific recovery time has not yet been determined.
• FreeBSD 9.1-RELEASE will be officially released after the package preparation.
• Some committer's access to the FreeBSD.org cluster may be affected. At present, the identification of personal identity is largely dependent on PGP signatures rather than personal identity documents, and the relevant recovery operations have been carried out for quite a long time.

The lessons I learned from this incident are:

• Setting an effective intrusion detection system is very important. Timely intrusion detection can effectively prevent further damage (FreeBSD-CURRENT provides a new distributed audit mechanism auditstd, can be used as an auxiliary means of intrusion detection );
• Existing systems with MD5 should be upgraded to SHA512 or Blowfish;
• Avoid using weak passwords and use the same password in different places;
• Encrypt the SSH private key and use a strong password;
• Use different SSH private keys based on the role. If the SSH authentication agent is used, the private key should expire within a certain period of time;
• The correct implementation of the dual authentication can help to delay the pace of attackers and enable the intrusion detection system to detect problems in a timely manner;
• Adequate offline backup is very important;