Checking For A DoS

Working on groups of web servers, especially ones that are highly susceptible to attack, it is a good idea to have a string of commands that will allow you to check what is going on.

Check for DDos:

netstat -n | grep EST | awk '{ print $5 }' | cut -d: -f1 | sort | uniq -c | sort -nr | perl -an -e 'use Socket; ($hostname, @trash) = gethostbyaddr(inet_aton($F[1]), AF_INET); print "$F[0]\t$F[1]\t$hostname\n";'

Using this command will produce a list of hostnames that have a connect to the machine in an ESTABLISHED state. This is handy for creating a firewall rule either on the host (iptables, ipfw) or a little further away from the machine (at the edge router).

Check for web attacks:

cat eric.lubow.org-access_log.20081015 | awk '{print $1 }' | sort | uniq -c | sort -nr | head | perl -an -e 'use Socket; ($hostname, @trash) = gethostbyaddr(inet_aton($F[1]), AF_INET); print "$F[0]\t$F[1]\t$hostname\n";'

By using this command, you will get a hostname lookup on the IP sorted by total hit count descending. As when checking for DDos attacks, you can use this information to write firewall rules.

More web attack checks:

for i in `ls *.20081015 | grep -v error`; do echo "##### $i ######"; tail -n 10000 $i| awk '{print $1};' | sort -n | uniq -c | sort -nr | head -2; done

The difference between this check and the previous check is that this time, you may have a lot more logfiles to go through. I am also assuming that they are stored by .. They will print out which file its scanning and the top 2 issues from that file.

Referrer Check:

for file in `ls -lrS *access*20080525* | tail -n20`; do echo "==========" $file; gawk --re-interval -F'"' '{ split($4, myrt, "/");  split($0, myct); split(myct[3], myc, " "); if (length(myrt[3])==0) { myrt[3]="none"}; if (myrt[3] ~ /([[:digit:]]{1,3}\.){3}[[:digit:]]{1,3}/) { referrers[myrt[3]"/"myc[1]]++; } else { t=split(myrt[3], myrt2, "."); myref="*."myrt2[t-1]"."myrt2[t]; referrers[myref"/"myc[1]]++; } } END { for (referrer in referrers) { print referrers[referrer], referrer } }' $file | grep -v none | sort -n; done

This last check will get the referrer for a page from the logs and count up the number of times that exact referrer drives traffic to your page. Although this may initially appear to be only tangentially useful, if you are getting DDos, it may be hard to track down. Let’s say that you have some static content like a funny image and want to know why everyone is going to that image. Maybe your getting Dugg or ./ and this will help you tell (and find out what your page is so you can Digg yourself if you’re into that sort of thing).

SSH Organization Tips

Over the years, I have worked with many SSH boxen and had the pleasure to manage even more SSH keys. The problem with all that is the keys start to build up and then you wonder which boxes have which keys in the authorized keys file and so on and so on. Well, I can’t say I have the ultimate solution, but I do have a few tips that I have come across along the way. Hopefully they will be of use to someone else besides myself.

  1. Although this should hopefully already be done (my fingers are crossed for you), check the permissions on your ~/.ssh directory and the file contained in it.
    $ chmod 700 ~/.ssh
    $ chmod 600 ~/.ssh/id_dsa
    $ chmod 640 ~/.ssh/id_dsa.pub
  2. Now that SSHv2 is pretty widely accepted, try using that for all your servers. If that isn’t possible, then try to use SSHv2 whenever possible. This means a few things.
    1. Change your /etc/ssh/sshd_config file to say:
      Protocol 2

      instead of

      Protocol 1
    2. Don’t generate anymore RSA keys for yourself. Stick to the DSA keys:
      $ cd ~/.ssh
      $ ssh-keygen -t dsa
    3. Use public key based authentication and not password authentication. To do this change your /etc/ssh/sshd_config file to read:
      PubkeyAuthentication yes

      instead of

      PubkeyAuthentication no
  3. Keeping track of which keys are on the machine is a fairly simple yet often incomplete task. To allow for a user to login using their SSH(v2) key, we just add their public key to the ~/.ssh/authorized_keys file on the remote machine:
    1. Copy the file to the remote machine:
      $ scp id_dsa.pub user@host:.ssh/
    2. Append the key onto the authorized_keys file:
      $ cat ~/.ssh/id_dsa.pub >> ~/.ssh/authorized_keys

    Before moving on here and just deleting the public key, let’s try some organizational techniques.

    1. Create a directory in ~/.ssh to store the public keys in:
      $ mkdir ~/.ssh/pub
    2. Move the public key file into that directory and change the name to something useful:
      $ mv ~/.ssh/id_dsa.pub ~/.ssh/pub/root@main.mydomain.com
    3. NOTE: Don’t do this unless you are sure that you can log in with your public key otherwise you WILL lock yourself out of your own machine.

  4. Now a little of the reverse side of this. If a public key is no longer is use, then you should remove it from your ~/.ssh/authorized_keys file. If you have been keeping a file list in the directory, then the file should be removed from the directory tree as well. A little housekeeping is not only good for security, but also some piece of mind in efficiency and general cleanliness.
  5. Although this last item isn’t really organizational, it is just really handy and I will categorize it under the title of efficiency. Using ssh-agent to ssh around. If you are a sysadmin and you want to only type your passphrase once when you login to your computer, then do the following:
    1. Check to see if the agent is running:
      $ ssh-add -L

      NOTE: If ssh-agent is not running, it will say The agent has no identities.

    2. If its running, continue to the next step, otherwise type:
      $ ssh-agent
    3. Now to add your key to the agent’s keyring, type:
      $ ssh-add

    SSH to a machine that you know has that key and you will notice that you will no longer have to type in your passphrase while your current session is active.

These are just some tricks that I use to keep things sane. They may not work for you, but some of them are good habits to get into. Good luck.

10 More Tips Towards Securing Your Linux System

Since everyone seemed to enjoy my first round of tips and tricks to securing a linux system, I figured I would throw together a few more. Enjoy.

  1. There are files that get changed very infrequently. For instance, if your system won’t have any users added anytime soon then it may be sensible to chattr immutably the /etc/password and /etc/shadow files. Only the superuser or a process possessing the CAP_LINUX_IMMUTABLE capability can set or clear this attribute.
       chattr +i /etc/passwd /etc/shadow
  2. Password protect your linux install with LILO. Edit your /etc/lilo.conf. At the end of each linux image that you want to secure, put the lines:
       read-only
       restricted
       password = MySecurePassword

    Ensure you rereun /sbin/lilo so the changes take effect.

  3. Users who have sudoer (sudo) accounts setup can have the account setup to change to root without a password. To check this, as root use the following command:
       grep NOPASSWD /etc/sudoers

    If there is an entry in the sudoers file, it will look like this:

       eric    ALL=NOPASSWD:ALL

    To get rid of this, type visudo and remove the line in that file.

  4. Use sudo to execute commands as root as a replacement for su. In the /etc/sudoers file, add the following lines by using the visudo command:
       Cmnd_Alias LPCMDS = /usr/sbin/lpc, /usr/bin/lprm
       eric    ALL=LPCMDS

    Now the user ‘eric’ can sudo and use the lpc and lprm commands without having any other root level access.

  5. Turn off PasswordAuthentication and PermitEmptyPasswords in the SSH configuaration file /etc/ssh/sshd_config. This will ensure that users cannot set empty passwords or login without SSH keys.
       PermitEmptyPasswords no
       PasswordAuthentication no
  6. Instead of using “xhost +” to open up access to the X server, be more specific. Use the server name that you are allowing control to:
       xhost +storm:0.0

    Once you are done using it, remember to disallow access to the X server from that host:

       xhost -storm:0.0
  7. To find out the .Xauthority magic cookie looks like and to send it (authorization information) to the remote host, use the following command:
       xauth extract - $DISPLAY | ssh storm xauth merge -

    Now the user who ran this command on the original host can now run xcilents on storm. xauth needs to be present on both hosts.

  8. To turn on spoof protection, run a simple bash script:
       for i in /proc/sys/net/ipv4/conf/*/rp_filter; do echo 1 > $i done;

    Be careful to remember that it drops packets more or less ‘invisibly’.

  9. A SYN-flood attack has the ability to bring the network aspect of your linux box to a snail like crawl. TCP_SYNCookies protection attempts to stop this from taking a heavy toll on the machine. To enable cp_syncookies
    protection, use the following command:

       echo 1 > /proc/sys/net/ipv4/tcp_syncookies
  10. When possible use secure connection methods as opposed to insecure methods. Unless you are required to use telnet, substitute ssh (Secure SHell) in for rsh or telnet. Instead of POP3 or IMAP use SPOP3 or SIMAP (IMAPS). Both SIMAP and SPOP3 are just versions of IMAP and POP3 running over an SSL (Secure Socket Layer) tunnel.

10 Tips To Start Securing Your Linux System

A while back I had been asked to write a few quick tips that as an administrator, one would find helpful. They published in one form or another and are now available here. There are MANY more, but these are just a few. Enjoy.

  1. Users who may be acting up or aren’t listening can still be controlled. Using a program called ‘skill’ (signal kill) which is part of the ‘procps’ package.
       Halt/Stop User eric: skill -STOP -u eric
       Continue User eric: skill -CONT -u eric
       Kill and Logout User eric: skill -KILL -u eric
       Kill and Logout All Users: skill -KILL -v /dev/pts/*
  2. Make use of security tools out there to test your server’s weaknesses. Nmap is an excellent port scanning tool to test to see what ports you have open. On a remote machine, type the command:
       # nmap -sTU <server_ip>

       Starting nmap 3.70 ( http://www.insecure.org/nmap/ ) at 2006-08-10 13:51 EST
       Interesting ports on eric (172.16.0.1):
       (The 3131 ports scanned but not shown below are in state: closed)
       PORT    STATE         SERVICE
       22/tcp  open          ssh
       113/tcp open          auth
       
       Nmap run completed -- 1 IP address (1 host up) scanned in 221.669 seconds
  3. On a production server that is in a common area (although this should not be the case, some situations are inevidable). To avoid an accidental CTRL-ALT-DEL reboot of the machine, do the following to remove the necessary
    lines from the /etc/inittab file:

       # sed -i 's/ca::ctrlaltdel:/#ca::ctrlaltdel:/g' /etc/inittab
  4. Two SSH configuration options that can be set to improve security should be checked on your production server. UsePrivilegeSeparation is an option, when enabled will allow the OpenSSH server to run a small (necessary) amount of code as root and the of the code in a chroot jail environment. StrictModes checks to ensure that your ssh files and directories have the proper permissions and ownerships before allowing an SSH session to open up. The
    directives should be set in the /etc/ssh/sshd_config as follows:

       UsePrivilegeSeparation yes
       StrcitModes yes
  5. The default umask (usermask) on most systems should be 022 to ensure that files are created with the permissions 0644 (-rw-r–r–). To change the default umask setting for a system, edit /etc/profile to ensure that you umask is appropriate for your setup.
  6. Some users like to have a passwordless account. To check this you need to look at the /etc/shadow account with the following command line:
    awk -F: '$2 == "" { print $1, "has no password!" }' /etc/shadow
  7. Just in case someone else who has access to the superuser account decided to alter the password file and potentially make themselves a superuser. This is a method to check:
       awk -F: '$3 == 0 { print $1, "is a superuser!" }' /etc/passwd
  8. Setuid and Setgid files have the potential to be very hazardous if they are accessilbe by the wrong users on the system. Therefore it is handy to be able to check with files fall into this category.
       find /dir -xdev -type f -perm +ug=s -print
  9. World writable files can be left around by users wanting to make things easier for themselves. It is necessary to be careful about who can write to which files. To find all world writable files:
       find /dir -xdev -perm +o=w ! \( -type d -perm +o=t \) ! -type l -print
  10. Some attackers, prior to attacking a host, (or users nmaping a host) will check to see if the host is alive. They do this by ‘ping’ing the host. In order to check if the host is up, they will use an ICMP echo request packet.
    To disallow these types of packets, use iptables:

       iptables -A INPUT -p icmp --icmp-type echo-request -j DROP