Archive for 2008

Checking For A DoS

Monday, November 17th, 2008

Working on groups of web servers, especially ones that are highly susceptible to attack, it is a good idea to have a string of commands that will allow you to check what is going on.

Check for DDos:

1
netstat -n | grep EST | awk '{ print $5 }' | cut -d: -f1 | sort | uniq -c | sort -nr | perl -an -e 'use Socket; ($hostname, @trash) = gethostbyaddr(inet_aton($F[1]), AF_INET); print "$F[0]\t$F[1]\t$hostname\n";'

Using this command will produce a list of hostnames that have a connect to the machine in an ESTABLISHED state. This is handy for creating a firewall rule either on the host (iptables, ipfw) or a little further away from the machine (at the edge router).

Check for web attacks:

1
cat eric.lubow.org-access_log.20081015 | awk '{print $1 }' | sort | uniq -c | sort -nr | head | perl -an -e 'use Socket; ($hostname, @trash) = gethostbyaddr(inet_aton($F[1]), AF_INET); print "$F[0]\t$F[1]\t$hostname\n";'

By using this command, you will get a hostname lookup on the IP sorted by total hit count descending. As when checking for DDos attacks, you can use this information to write firewall rules.

More web attack checks:

1
for i in `ls *.20081015 | grep -v error`; do echo "##### $i ######"; tail -n 10000 $i| awk '{print $1};' | sort -n | uniq -c | sort -nr | head -2; done

The difference between this check and the previous check is that this time, you may have a lot more logfiles to go through. I am also assuming that they are stored by .. They will print out which file its scanning and the top 2 issues from that file.

Referrer Check:

1
for file in `ls -lrS *access*20080525* | tail -n20`; do echo "==========" $file; gawk --re-interval -F'"' '{ split($4, myrt, "/");  split($0, myct); split(myct[3], myc, " "); if (length(myrt[3])==0) { myrt[3]="none"}; if (myrt[3] ~ /([[:digit:]]{1,3}\.){3}[[:digit:]]{1,3}/) { referrers[myrt[3]"/"myc[1]]++; } else { t=split(myrt[3], myrt2, "."); myref="*."myrt2[t-1]"."myrt2[t]; referrers[myref"/"myc[1]]++; } } END { for (referrer in referrers) { print referrers[referrer], referrer } }' $file | grep -v none | sort -n; done

This last check will get the referrer for a page from the logs and count up the number of times that exact referrer drives traffic to your page. Although this may initially appear to be only tangentially useful, if you are getting DDos, it may be hard to track down. Let’s say that you have some static content like a funny image and want to know why everyone is going to that image. Maybe your getting Dugg or ./ and this will help you tell (and find out what your page is so you can Digg yourself if you’re into that sort of thing).

Adding Yum to CentOS 5

Thursday, October 30th, 2008

I use a lot of VPS and often times, they don’t actually have yum to make my life easier. So here is a quick HOWTO on installing yum on a CentOS box. This assumes that you have rpm and wget already installed. Note: This will only work on CentOS 5.2 while the mirror is still active.

Run the following code in a temporary directory to download all the RPMs.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
#!/bin/bash

for file in \
        elfutils-0.125-3.el5.i386.rpm \
        elfutils-libs-0.125-3.el5.i386.rpm \
        expat-1.95.8-8.2.1.i386.rpm \
        gmp-4.1.4-10.el5.i386.rpm \
        libxml2-2.6.26-2.1.2.1.i386.rpm \
        libxml2-python-2.6.26-2.1.2.1.i386.rpm \
        m2crypto-0.16-6.el5.2.i386.rpm \
        python-2.4.3-21.el5.i386.rpm \
        python-elementtree-1.2.6-5.i386.rpm \
        python-iniparse-0.2.3-4.el5.noarch.rpm \
        python-sqlite-1.1.7-1.2.1.i386.rpm \
        python-urlgrabber-3.1.0-2.noarch.rpm \
        readline-5.1-1.1.i386.rpm \
        rpm-4.4.2-48.el5.i386.rpm \
        rpm-libs-4.4.2-48.el5.i386.rpm \
        rpm-python-4.4.2-48.el5.i386.rpm \
        sqlite-3.3.6-2.i386.rpm \
        yum-3.2.8-9.el5.centos.1.noarch.rpm \
        yum-metadata-parser-1.1.2-2.el5.i386.rpm
  do wget http://mirror.centos.org/centos-5/5.2/os/i386/CentOS/$file;
done

Once you have downloaded the necessary files. Install them all by typing:

1
# rpm -Uvh *.rpm

Then feel free to # yum -y update to bring your system up to date.

Character Encoding

Thursday, October 23rd, 2008

I recently ran into some character encoding issues and I wanted to share the fun. The default character encoding for MySQL on Gentoo is latin-1 or iso-8859-1. This wasn’t a problem until we recently started putting content straight from the DB through Java and onto the web. Java connects to the DB with a character encoding (typically UTF-8). Since UTF-8 is roughly a superset of iso-8859-1, it generally wasn’t a problem. Except when UTF-8 and UTF-16 characters were put into an iso-8859-1 database without translation.

What was essentially happening was that the data was being stored as iso-8859-1. The Java code was connecting to the DB in UTF-8 and pulling it into Java (which is usually UTF-16, but in this case was being handled as UTF-8). It was then being sent to the browser as URL encoded UTF-8 when in reality, it hadn’t even properly been put into UTF-8 character encoding. This then gave the web browser some funny yen symbols and question marks. This was not quite what we were aiming for.

The moral of this story is that it is necessary to realize the character encoding of the start point and end point of your data. It is crucial that the code points match up otherwise they could potentially make for an interesting screen given to the reader. All this could have been avoided with a simple: ALTER TABLE myTable MODIFY myColumn VARCHAR(255) CHARACTER SET utf8;.

MySQL Encoded URI Search and Replace

Friday, September 26th, 2008

So I can definitely not take credit for this trick. The credit goes to Augusto Bott of the Pythian Group for this. I have a table in my DB that has columns of encoded URIs. The list of URI encoding character translations is available here.

To accomplish this, the MySQL REPLACE function was used.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
mysql> SET @test:='%20%27%7C%26%5E%2B%2D%25';
Query OK, 0 rows affected (0.00 sec)

mysql> SELECT @test as original,
    ->        REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(
    ->        @test,'%20',' '),
    ->        '%27','\''),
    ->        '%7C','|'),  -- REPLACE() is case sensitive
    ->        '%7c','|'),  -- so we have
    ->        '%26','&'),
    ->        '%5E','^'),
    ->        '%5e','^'),  -- to repeat
    ->        '%2D','-'),
    ->        '%2d','-'),  -- ourselves
    ->        '%2B','+'),
    ->        '%2b','+'),  -- sometimes
    ->        '%25','%') as replaced;
+--------------------------+----------+
| original                 | replaced |
+--------------------------+----------+
| %20%27%7C%26%5E%2B%2D%25 |  '|&^+-% |
+--------------------------+----------+
1 row in set (0.01 sec)

To make our lives easier, let’s create a function from that.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
mysql> CREATE FUNCTION url_replace
    -> (url VARCHAR(1024))
    -> RETURNS VARCHAR(1025)
    -> BEGIN
    -> RETURN REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(
    ->        @test,'%20',' '),
    ->        '%27','\''),
    ->        '%7C','|'),
    ->        '%7c','|'),
    ->        '%26','&'),
    ->        '%5E','^'),
    ->        '%5e','^'),
    ->        '%2D','-'),
    ->        '%2d','-'),
    ->        '%2B','+'),
    ->        '%2b','+'),
    ->        '%25','%');
    -> END$$
Query OK, 0 rows affected (0.02 sec)

And to wrap it up, let’s use it in an actual query. This query finds all instances of a % and replaces it with its unencoded character equivalent.

1
mysql> UPDATE website_page SET uri = REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE( uri,'%20',' '), '%27','\''), '%7C','|'), '%7c','|'), '%26','&'), '%5E','^'), '%5e','^'), '%2D','-'), '%2d','-'), '%2B','+'), '%2b','+'), '%25','%'), '%2C',',') WHERE uri LIKE '%\%%';

Apache mod_proxy

Tuesday, September 16th, 2008

I came up against the interesting problem of putting multiple stand alone apache tomcat instances with different virtual host names on the same machine that all needed to be accessible via port 80 (on the same IP). There is always mod_jk, but that seems like a bit too much to fix a simple problem. Being a strong believer in the right tool for the right job, I came across mod_proxy. This way I get to take advantage of apache connection handling without having to put a whole proxy server in front of it. Because there is dispatching by virtual host to do, putting apache in front just seemed to be the best idea.

Since there aren’t too many clear HOWTOs on this, it took a bit of fudging. Here is what you need to know.

Let’s create the host http://port8080.lubow.org/ to go to http://8080.lubow.org:8080/.

The first thing is a fairly common default configuration of NameVirtualHost option. This is so you can have multiple virtual hosts per IP. Unless you are crazy (or have a really good reason), you do not want to create an open proxy. So you need to globally configure the ProxyRequests variable to be off. Do the base setup for a VirtualHost of ServerName and ServerAdmin.

Setup the proxy authorizations (similar to the apache allow/denys). In order for the right HTTP headers to make it to the proxy’d virtual host, the headers will need to be rewritten. This needs to happen both going to the host and coming back from the host going to the client. This is why there is the ProxyPass and ProxyPassReverse. The first argument is the URL that on the virtual host that should match the URL (second argument) on the proxy’d virtual host. The ProxyPreserveHost option is generally not needed (but it is for the specific application I am running. Click the link above to read the description to determine whether it is right for you.

Putting it all together, you will get a file that looks like below. Make sure you replace your IPs and hostnames with what’s appropriate for your environment.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
ProxyRequests Off
NameVirtualHost 1.2.3.4:80

<virtualhost 1.2.3.4:80>
     ServerAdmin webmaster@lubow.org
         ServerName port8080.lubow.org
         <proxy *>
                  Order deny,allow
                  Allow from all
         </proxy>
         ProxyPreserveHost   On
         ProxyPass   / http://8080.lubow.org:8080/
         ProxyPassReverse    / http://8080.lubow.org:8080/
</virtualhost>

The Next Step In Browser Evolution

Tuesday, September 2nd, 2008

I was having a chat with my two friends from Redub Consulting about the new Google Chrome browser. At a cursory exploration, we found that (as promised) the Javascript engine is incredibly fast. But I don’t want to throw that out there since Google already us that in their Chrome Comic. I want to talk about where this could be leading.

As some of you know, Adobe Air is a desktop application that can interact with internet applications. The catch here is that since its a desktop application, it has access to the same elements of the physical machine as any other desktop application (USB ports, printers, sound/video out ports, etc). Browsers don’t yet that kind of access to a computer. They are limited to the user space in which they are run in. All the sound and video you hear and see is sent through 3rd party applications within the browser. What if the browser could control those elements of your machine? What if your entire computer experience was now internet based. Google is already trying to push this with software as a service (GoogleDocs), but keep extending this idea. What if your media center could be controlled via an internet application?

Eclipse IDE is now at a point at which you can your code as its running and change function calls at the opcode level to avoid recompiling your program over and over. Eclipse has grown to the point where its almost like an OS in its capabilities. In that same vein, Google’s new browser now controls its individual tabs and sandboxes each tab in order to have task level control over potentially runaway web applications.

So what I am trying to say here? I’m glad you asked. I believe this browser is the next step towards ubiquitous computing in the sense that 1 application to control your internet (or whole user) experience. AppleTV for instance is a set of specially designed hardware that can be interacted with over the internet. By allowing applications, such as Air (and potentially soon Chrome) to internet directly with the hardware attached to the computer, you are are negating the need for that specially designed hardware. One piece of hardware can be designed to do it all in terms of the interactive experience. Google is stepping to the plate and pushing forward for just this type of innovation. Keep an eye on the features of Google Chrome to come. If it becomes integrated any deeper into the desktop, it will open up a new age of ubiquitous computing.

1 Extension, Multiple Phones

Tuesday, January 15th, 2008

In order to setup Asterisk to ring multiple phones from the same dialed extension, you will need to create a phantom extension. I accomplished this by doing the following…

Before we go any further, let’s use the following information as true. The extension we want to have ring in multiple places is extension 100. For sanity’s sake, let’s say we want it to ring in 3 places (regardless of the reason). This means that each phone will need its own extension and auth information according to the sip.conf.

First you need to assign each device (phone), its own extension. Let’s give each device the extension of <ext><n>. Therefore our 3 phones will have the extensions of 1001,1002,1003 respectively. Their entries in the sip.conf will look like this:

[1001]
type=peer
context=internal
username=1001
callerid=Eric Lubow <100>
host=dynamic
auth=1001@192.168.1.2
call-limit=100
nat=no
canreinvite=yes
mailbox=100@allstaff
disallow=all
allow=gsm
allow=ulaw
astdb=chan2ext/SIP/1001=1001

[1002]
type=peer
context=internal
username=1002
callerid=Eric Lubow <100>
host=dynamic
auth=1002@192.168.1.2
call-limit=100
nat=no
canreinvite=yes
mailbox=100@allstaff
disallow=all
allow=gsm
allow=ulaw
astdb=chan2ext/SIP/1002=1002

[1003]
type=peer
context=internal
username=1003
callerid=Eric Lubow <100>
host=dynamic
auth=1003@192.168.1.2
call-limit=100
nat=no
canreinvite=yes
mailbox=100@allstaff
disallow=all
allow=gsm
allow=ulaw
astdb=chan2ext/SIP/1003=1003

Next, in your extensions.conf, add the entry to ring all the extensions when the phantom extension is dialed. The Dial() command should now look as follows:

exten => 100,1,Dial(SIP/1001&SIP/1002&SIP1003,18)

A nice thing to do to (in order to not confuse the user) is, in your tftp files, ensure that the label on the phone (each phone) is still the actual extension of the phone that one would dial to get to it. Label the phone elsewhere with your REAL extension to keep track of it.