This document explains the basic steps you should take to recover after an intrusion has been discovered. If you are a College of Engineering system's administrator the exact procedure can be found here. (A printable checklist on this information is available here to aid you in tracking your recovery effort.)

1. Send Notification About the Incident

Send an initial notification that there was a compromise discovered on your system to the appropriate people. This is done so that other system administrators know that there was a compromise discovered and that they should check their systems as well.

Volatile information encompasses data which would disappear if the system in question was rebooted or taken off the network. It includes such data as running processes, open network connections and network interfaces in promiscuous mode.

Detailed information encompasses any information you can find about the intruder. This evidence is gathered into a report so that it can be used to notify other system administrators about the intruder's activity and about other possible break-ins by the intruder.

Send a detailed report about the intruder's activity on your system to the appropriate people. This is done so that messages can be sent to hosts the intruder came from to warn them about possible compromises.

Correct any changes the intruder made and close any security holes on your system to protect against further compromises.

 

SEND NOTIFICATION ABOUT THE INCIDENT

1. Gather preliminary information. Gather preliminary information about the intrusion to include in the notification. This would include information easily accessible by finding hacker files and browsing the last and system logs, such as: When the intruder first gained access to the system Where the intruder came from What the intruder was doing on the system How the intruder gained access to the system If a packet sniffer was found Do not spend an undo amount of time trying to figure out each detail. It is more important at this time to get a quick impression about the intruder so you can notify other system administrators about what to look for. 2. Notify MIS Security and your departmental security contact of the intrusion. Due to the importance of security at the University of Florida and the need for quick action to security problems, you should send this notification by the end of the day that the intruder was discovered. Use the Preliminary Report Template as a guide when sending the notification. If you can't gather all the information which is contained in it, send the notification without it.

 

GATHER VOLATILE INFORMATION

1. Check for anomalous processes left running by the hacker. From a process list (use the ps command), look for processes (particularly those running with root privledges) which stand out either because (1) you don't know what the process is/does or (2) it has unusual statistics (high CPU usage or been running for a long time). Typical tools left running by a hacker can include network/packet sniffers, Denial of Service (DoS) attacks and/or Internet Relay Chat (IRC) bots. Also, telnetd or ftpd processes which have been running for several days have been linked to hackers. We have seen packet sniffers named 'sniffit' and 'sniff', DoS attacks named 'bmb', 'teardrop', and 'smurf', and IRC bots named 'eggdrop' and 'bitchx'. However, it is not good enough to look for these names, or other unrecognized names, in a process listing since it is common practice to rename tools and exploits to common system binary names. 2. Check for open network connections. Using the netstat command examine the system for open network connections. This will show any open network connections (including source and destination host and socket number). The socket number can tell you what kind of process is running since most internet applications use standard ports. If processes are using port 6666 or 6667, these are active IRC sessions. 3. Check for network interfaces in promiscuous mode. One of the more important processes to search for is a network/packet sniffer. These tools typically capture username and password information from network traffic (not only traffic to/from the compromised machine but to/from any machine on the network wire). The primary step to tracking down if a sniffer was installed on your system is to see if any network interfaces are in promiscuous mode. Use a combination of netstat and ifconfig to determine what network interfaces you have and what flags are active. Flags such as ALLMULTI or PROMISC indicate an interface in promiscuous mode. You can also download and install ifstatus which is a tool designed specifically to see if network interfaces are in promiscous mode. If you find that a packet sniffer has been installed on your systems, we strongly urge you to examine the output file(s) to determine what other machines are at risk. In some cases you are even able to see some of what the hacker did in the sniffer logs.

 

GATHER DETAILED INFORMATION

1. Check the syslogs in /var/log and/or /var/adm for abnormal activity. Reviewing your log files will help you get a better idea of how your machine was compromised, what happened during the compromise and what remote hosts accessed your machine. Keep in mind, though, that when reviewing any log files from a root compromised machinev that any of the logs could have been modified by the intruder. As you browse through the syslog, messages, and security log files inspect anything that is out of the ordinary. Also, look for events that occurred around the times of the known intrusion. 2. Check webserver logs (access_log and error_log) for abnormal activity. Review web server log files (if there is a web server running) to determine if the web server is the source of the compromise. There are common exploits in /cgi-bin/handler, /cgi-bin/phf, and /cgi-bin/test-cgi to watch out for. 3. Check lastlog for suspicious logins. The lastlog is available through the use of the last command. It converts the binary information contained in /var/log/wtmp to text for viewing. This logfile contains information on each successfull user login/logout. Checking this file for suspicious connections (e.g., from unauthorized hosts) may be useful in determining other hosts that may have been involved and finding what accounts on your system may have been compromised. In addition to the information in the lastlog, you can check who is currently logged into your system using the who command. This information is stored in /var/log/utmp. 4. Check xferlog for suspicious file transfers. If the compromised system has a functioning ftp server, the xferlog contains a log for all ftp transfers. This may help you discover what intruder tools have been uploaded to your system, as well as what information has been downloaded from your system. 5. Check .history files for evidence of the hacker. Some command shells store each command that is executed at the command prompt. This information is usually contained in the file named .history located in the home directory of each user. Sometimes the hacker fails to realize this or fails to delete the file and leaves a trace of what he/she was doing on the system. Examine the .history files of accounts you think may have been compromised for evidence of hacker activity. 6. Check for modifications to the password file. Sometimes intruders create dummy accounts for themselves or modify system or user accounts. Thus, not only should you look for accounts which were added but you should look for: accounts with no passwords accounts with GIDs changed (typically to 0 (superuser)) accounts with changed shells etc. 7. Check for modifications to /etc/inetd.conf. Sometimes hackers install services as backdoors into the system. Check for any unusual additions or subtracts from /etc/inetd.conf. We have seen incidents where hackers have enabled services that previously had been thought turned office. 8. Check for modifications to 'cron' and 'at'. Cron and at can be used to start and stop processes at certain times. Check these to see if changes have been made. In addition, check to see if any of the jobs are world-writable. 9. Check for modifications to /etc/hosts.equiv and to users .rhosts files. These files specify hosts which can be trusted. (Read the hosts.equiv, .rhosts man page.) Examine these files for '+' entries and inappropriate non-local hosts. In addition, these files should not be world-writable. 10. Check for modifications to system binaries. A common practice for hackers is to replace system binaries with ones of their own which either contain backdoors into the system or hide the hackers activity on the system. We've seen intruders change programs such as login, su, telnet, netstat, ifconfig, ls, find, etc. Compare the binaries on the compromised system with known good copies, such as those from the distribution media. In addition you can use cryptographic checksum tools such as cmp, MD5 or Tripwire to compare binaries from day to day. 11. Check for files left by the hacker. Look everywhere on the system for unusual or hidden files (files which start with a period and are not normally shown by 'ls') because they are usually used to hide tools and information. A common practice is to create hidden directories like '...' or '.. ' (dot dot space) in user directories to store hacker tools and files. In addition to, files with names such as '.xx' and '.mail' have been used (that is, files that may appear as normal). Finally, another common practice is to store unexpected ASCII files and directories in the /dev directory. The find program can be used to find these types of files: find / -name ".*" -print -xdev 12. Check for SETUID and SETGID scripts. Search everywhere for setuid and setgid files (especially setuid root). Intruders often leave setuid root executables or shells to allow them root access at a later time. The find command can be used to find setuid root files: find / -user root -perm -4000 -print 13. Check other computer systems on your network. Examine the other computer systems on your network to determine if the intruder was able to compromise them. If you find evidence of a hacker on them, or you find a network sniffer on any compromised host, proceed with this checklist on each host. Most of the time, if one host has been compromised, others on the network have been too. This is especially tru for networks where NIS is running or where hosts trust each other through the use or .rhosts or /etc/hosts.equiv files.

 

REPORT THE INCIDENT

1. Send a report of your findings to MIS Security, security@eng.ufl.edu, and to your departmental security contact After you have thoroughly investigated this incident send a report of you findings to MIS Security. The report should contain the following: Date the intrusion was discovered How the intrusion was discovered Date the hacker first gained entry into the system Exploit used to gain entrance to the system (if found) If there was a root compromise If there was a sniffer installed If modified system binaries were found A listing of compromised hosts (including name, ip #, OS (version), and time information). A listing of hosts the hacker came from. What hacker tools/files where found. What the hacker was believed to be doing. Log entries from the lastlog and syslogs which support claims. You can use the Intruder Report Template as a sample... just fill in the blanks and email it.

 

SECURE YOUR SYSTEM(S)

1. Verify the integrity of the operating system. Correct any changes that the intruder made to your system. In some instances a complete reinstall of the operating system is the best choice. While determining the best course of action keep in mind that if a machine is root compromised, anything on that machine could be modified, including the kernel, binaries, and datafiles. 2. Apply operating system patches. A major step in defending your systems from attack is to apply the full set of security patches for your operating system to the compromised machine. It is also recommended that you check with your vendor regularly for updates or new patches. 3. Disable unnecessary services. Configure your system to offer only the services that the system is intended to offer, and no others. Check to ensure that there are no weaknesses in the configuration files for those services, and that those services are available only to the intended set of other systems. In general, the most conservative policy is to start by disabling everything and only enabling services as they are needed. 4. Apply patches to and/or upgrade necessary services. In addition to disabling services, you should apply security patches and/or upgrades to services that you need to have enabled. Like operating system patches, you should check regularly for patches/upgrades to these services that affect the security of your system. 5. Check for CERT advisories. Consult past CERT advisories, summaries, and vendor-initiated bulletins, and follow the instructions relevant to your situation. Be sure to apply all patches and/or workarounds that are mentioned in them. Past CERT advisories are found at: ftp://ftp.cert.org/pub/cert_advisories Past CERT summaries are available at: ftp://ftp.cert.org/pub/cert_summaries Vendor-initiated bulletins are available at: ftp://ftp.cert.org/pub/cert_bulletins 6. Change all users passwords. After all security holes or configuration problems have been patched or corrected, you should change passwords to all accounts on the compromised systems. Ensure that passwords for all accounts are not easy to guess. You may want to consider using vendor-supplied or third-party tools to enforce your password policies. 7. Have a security scan done. To analyze the integrity of your system and the rest of your subnet a security scan should be done. Currently NERDC network services uses a software package from Internet Security Systems (ISS) to scan computer systems for known security holes, configuration problems and easily guessable passwords. This scan produces a report of it's findings and gives guidelines for further securing your systems. If you are the subnet manager, notify security@eng.ufl.edu that you want a security scan of your subnet done. If you are not the subnet manager, notify your subnet manager that you have recently recovered from a hacker compromise and would like a security scan done. Once you receive the results of the security scan, follow the instructions in the report to correct any security holes. If there were significant changes to be made, have a followup scan done after you make the changes to see if any new problems are found