Archive for January, 2007

10 More Tips Towards Securing Your Linux System

Wednesday, January 31st, 2007

Since everyone seemed to enjoy my first round of tips and tricks to securing a linux system, I figured I would throw together a few more. Enjoy.

  1. There are files that get changed very infrequently. For instance, if your system won’t have any users added anytime soon then it may be sensible to chattr immutably the /etc/password and /etc/shadow files. Only the superuser or a process possessing the CAP_LINUX_IMMUTABLE capability can set or clear this attribute.
    1
       chattr +i /etc/passwd /etc/shadow
  2. Password protect your linux install with LILO. Edit your /etc/lilo.conf. At the end of each linux image that you want to secure, put the lines:
    1
    2
    3
       read-only
       restricted
       password = MySecurePassword

    Ensure you rereun /sbin/lilo so the changes take effect.

  3. Users who have sudoer (sudo) accounts setup can have the account setup to change to root without a password. To check this, as root use the following command:
    1
       grep NOPASSWD /etc/sudoers

    If there is an entry in the sudoers file, it will look like this:

    1
       eric    ALL=NOPASSWD:ALL

    To get rid of this, type visudo and remove the line in that file.

  4. Use sudo to execute commands as root as a replacement for su. In the /etc/sudoers file, add the following lines by using the visudo command:
    1
    2
       Cmnd_Alias LPCMDS = /usr/sbin/lpc, /usr/bin/lprm
       eric    ALL=LPCMDS

    Now the user ‘eric’ can sudo and use the lpc and lprm commands without having any other root level access.

  5. Turn off PasswordAuthentication and PermitEmptyPasswords in the SSH configuaration file /etc/ssh/sshd_config. This will ensure that users cannot set empty passwords or login without SSH keys.
    1
    2
       PermitEmptyPasswords no
       PasswordAuthentication no
  6. Instead of using “xhost +” to open up access to the X server, be more specific. Use the server name that you are allowing control to:
    1
       xhost +storm:0.0

    Once you are done using it, remember to disallow access to the X server from that host:

    1
       xhost -storm:0.0
  7. To find out the .Xauthority magic cookie looks like and to send it (authorization information) to the remote host, use the following command:
    1
       xauth extract - $DISPLAY | ssh storm xauth merge -

    Now the user who ran this command on the original host can now run xcilents on storm. xauth needs to be present on both hosts.

  8. To turn on spoof protection, run a simple bash script:
    1
       for i in /proc/sys/net/ipv4/conf/*/rp_filter; do echo 1 > $i done;

    Be careful to remember that it drops packets more or less ‘invisibly’.

  9. A SYN-flood attack has the ability to bring the network aspect of your linux box to a snail like crawl. TCP_SYNCookies protection attempts to stop this from taking a heavy toll on the machine. To enable cp_syncookies
    protection, use the following command:

    1
       echo 1 > /proc/sys/net/ipv4/tcp_syncookies
  10. When possible use secure connection methods as opposed to insecure methods. Unless you are required to use telnet, substitute ssh (Secure SHell) in for rsh or telnet. Instead of POP3 or IMAP use SPOP3 or SIMAP (IMAPS). Both SIMAP and SPOP3 are just versions of IMAP and POP3 running over an SSL (Secure Socket Layer) tunnel.

10 Tips To Start Securing Your Linux System

Monday, January 29th, 2007

A while back I had been asked to write a few quick tips that as an administrator, one would find helpful. They published in one form or another and are now available here. There are MANY more, but these are just a few. Enjoy.

  1. Users who may be acting up or aren’t listening can still be controlled. Using a program called ‘skill’ (signal kill) which is part of the ‘procps’ package.
    1
    2
    3
    4
       Halt/Stop User eric: skill -STOP -u eric
       Continue User eric: skill -CONT -u eric
       Kill and Logout User eric: skill -KILL -u eric
       Kill and Logout All Users: skill -KILL -v /dev/pts/*
  2. Make use of security tools out there to test your server’s weaknesses. Nmap is an excellent port scanning tool to test to see what ports you have open. On a remote machine, type the command:
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
       # nmap -sTU <server_ip>

       Starting nmap 3.70 ( http://www.insecure.org/nmap/ ) at 2006-08-10 13:51 EST
       Interesting ports on eric (172.16.0.1):
       (The 3131 ports scanned but not shown below are in state: closed)
       PORT    STATE         SERVICE
       22/tcp  open          ssh
       113/tcp open          auth

       Nmap run completed -- 1 IP address (1 host up) scanned in 221.669 seconds
  3. On a production server that is in a common area (although this should not be the case, some situations are inevidable). To avoid an accidental CTRL-ALT-DEL reboot of the machine, do the following to remove the necessary
    lines from the /etc/inittab file:

    1
       # sed -i 's/ca::ctrlaltdel:/#ca::ctrlaltdel:/g' /etc/inittab
  4. Two SSH configuration options that can be set to improve security should be checked on your production server. UsePrivilegeSeparation is an option, when enabled will allow the OpenSSH server to run a small (necessary) amount of code as root and the of the code in a chroot jail environment. StrictModes checks to ensure that your ssh files and directories have the proper permissions and ownerships before allowing an SSH session to open up. The
    directives should be set in the /etc/ssh/sshd_config as follows:

    1
    2
       UsePrivilegeSeparation yes
       StrcitModes yes
  5. The default umask (usermask) on most systems should be 022 to ensure that files are created with the permissions 0644 (-rw-r–r–). To change the default umask setting for a system, edit /etc/profile to ensure that you umask is appropriate for your setup.
  6. Some users like to have a passwordless account. To check this you need to look at the /etc/shadow account with the following command line:
    1
    awk -F: '$2 == "" { print $1, "has no password!" }' /etc/shadow
  7. Just in case someone else who has access to the superuser account decided to alter the password file and potentially make themselves a superuser. This is a method to check:
    1
       awk -F: '$3 == 0 { print $1, "is a superuser!" }' /etc/passwd
  8. Setuid and Setgid files have the potential to be very hazardous if they are accessilbe by the wrong users on the system. Therefore it is handy to be able to check with files fall into this category.
    1
       find /dir -xdev -type f -perm +ug=s -print
  9. World writable files can be left around by users wanting to make things easier for themselves. It is necessary to be careful about who can write to which files. To find all world writable files:
    1
       find /dir -xdev -perm +o=w ! \( -type d -perm +o=t \) ! -type l -print
  10. Some attackers, prior to attacking a host, (or users nmaping a host) will check to see if the host is alive. They do this by ‘ping’ing the host. In order to check if the host is up, they will use an ICMP echo request packet.
    To disallow these types of packets, use iptables:

    1
       iptables -A INPUT -p icmp --icmp-type echo-request -j DROP

Patching Procedure vs. Exploitation Potential

Thursday, January 25th, 2007

When you talk to many security experts, they pretty much agree that when a vulnerability hits, that it’s necessary that it be patched and that its only a matter of time until the sh*t hits the fan and some real knowledgable black hat has put something together for the script kiddies to play with. But a lot of people seem to forget every time a patch is required on a production system that there is due process that system administrators must go through. One of the primary steps is simply evaluation.

The primary questions that needs to be evaluated are:

What is the likelihood of the vulnerability being exploited or the damage that could be caused if it is exploited?

vs.

How long will it take to apply the patch, test it, implement it, then deploy it to the production environment? What kind of impact will that have on the production servers in terms of outages/downtime? Will it break anything else?

Let’s take some time to break these down. I have always found that the easiest way for most people to understand a problem is to use an example. I don’t want to single out phpBB, but since it recently came up and spurred a necessary conversation, I will use it for my example. The advisory that I am referencing is available here from Bugtraq.

At one of the many websites I run, I administer a phpBB forum. The forum is relatively low volume, but high volume enough to attract spammers which means its likely that it also attracts hackers (of the black hat variety). The phpBB version is 2.0.21. For a few reasons, we have not only modified some of the source code of phpBB, but we have also added plugins. For anyone who has any experience adding plugins into phpBB, you know that its akin to chewing glass (to say the least). Even though we version track in CVS, it would still be somewhat of a PITA to update to 2.0.22. The process would be something along the lines of:

Import the new version into the old version with the changes into CVS. See if it makes sense to resolve the conflicts. If so, resolve the conflicts and begin testing. If not, figure out how to duplicate the changes in the previous version (2.0.21) in the new version (2.0.22). Once that’s been done, then add the plugins that were installed in the old version into the new version. Come up with a transition plan for the production server. Back up the data and do a few test runs of the transition on the development box. Then schedule the outage time and do the turnover to the new server. Then pray everything goes ok for the transition. Simple, No?

The point of going through that lengthy explanation was to demonstrate that the upgrade process may not be as simple (in a lot of cases) as:

1
apt-get update && apt-get upgrade

The exploit itself requires a user to create a shockwave flash file with certain parameters, then put it into a specific web page with certain parameters, and then it must be private messaged (emailed) to someone who is already signed into the board (has an active cookie).

Many security experts would tell you that, “It’s a vulnerability, it needs to be patched immediately.” Well, let’s do that evaluation thing I was referring to earlier. How likely is it that someone is going to take the time to create that flash file. And even if someone does go to that trouble, what’s to say that if a user (or the admin) receives the message in an email, that they are going to visit the site and watch the video?

My colleague was asserting that it’s out there on the internet and needs to be protected. And to that extent, I certainly agree. However, the amount of time that it would take to make all those changes, test them, and deploy the changes to the production server far outweighs the possibility of the application being exploited.

When I first started out in security, I took the approach, “It’s a vulnerability…Security at all costs.” Now I have learned that sometimes one needs to balance out time vs. need vs. priority. So I encourage System Administrators to think before jumping into situations like that. Think of how much more work could be accomplished in the time that would have been spent trying to patch something that probably wouldn’t have been exploited to begin with.

Configuring mod_security for EnGarde Secure Linux

Wednesday, January 24th, 2007

Introduction

This document is intended to guide a user through initially setting up and understanding a mod_security+Apache2 under EnGarde Secure Linux setup. Once you have completed reading this document, you should be able to understand the basics of mod_security, what it is used for, and why it may apply to you and your environment.

Why mod_security

The need for mod_security may not be initially apparent since we are all perfect programmers and rarely make a mistake that could prove hazardous to security. It may not be for you, but it is for the users of your servers who may not be as adept in creating web applications.

mod_security is a web application intrusion detection and prevention engine. It operates by ‘hook’ing itself into apache and inspecting all requests for your specific ruleset. It can be used to monitor your server with logging or even protect it by ”deny”ing attacks.

Skills Needed

You will need to have access to the WebTool and the GDSN Package Manager. You need to have shell access to the machine and the ability to use a text editor to make the necessary changes to the configuration files.

Installation

To install mod_security, go into the GDSN Manager in the Guardian Digital WebTool.

1
2
 System -> Guardian Digital Secure Network
 Module -> Package Management

Find the line that says libapache-mod_security and check the checkbox next to it. Click the Install Selected Packages button. Let the mod_security package install.

Configuration

Now its time to configure the mod_security package. The first thing that has to be done is to add the configuration file for mod_security (that we are going to create) to the apache2 configuration file. To accomplish this, ensure that the following line is somewhere in your /etc/httpd/conf/httpd.conf:

1
 Include conf/mod_security.conf

This ensures that when apache2 starts up, the configuration that you spcify in /etc/httpd/conf/httpd.conf will be loaded.

Basic Configuration

Once you have installed mod_security, it’s time for some basic configuration. In order to keep consistency, the mod_security.conf configuration file should be created in the /etc/httpd/conf/ directory. For a basic configuration (which we will walk through step-by-step), your /etc/httpd/conf/mod_security.conf file should looks as follows:

 LoadModule security_module /usr/libexec/apache/mod_security.so
 <IfModule mod_security.c>
   SecFilterEngine On
   SecFilterDefaultAction "log"
   SecFilterCheckURLEncoding On
   SecFilterForceByteRange 1 255

   SecServerSignature "Microsoft-IIS/5.0"

   SecAuditEngine RelevantOnly
   SecAuditLog /etc/httpd/logs/modsec_audit_log
   SecFilterDebugLog /etc/httpd/logs/modsec_debug_log
   SecFilterDebugLevel 0
 </IfModule>
SecFilterEngine

This directive turns on mod_security.

SecFilterDefaultAction

This directive decides what happens to a request that is caught by the mod_security filtering engine. In our case, we are going to log the request. By reading the documentation, you will find that there are many other options
available. By changing this line slightly (once you have logged and found out when and how the mod_security engine catches requests), you can deny requests and produce errors:

1
 SecFilterDefaultAction "deny,log,status:404"

This line denies the request, logs it to your log files, and send the requester back a HTTP status code 404 (also known as Page Not Found).

SecFilterCheckURLEncoding

This directive checks the URL to ensure that all characters in the URL are properly encoded.

SecFilterForceByteRange

This directive asserts which bytes are allowed in requests. The 1…255 specified in the example allows almost all characters. To bring this down to just the minimal ASCII character set, replace the above line with:

1
 SecFilterForceByteRange 32 126
SecServerSignature

This directive can be used to attempt to mask the identity of the apache server. Although this method works well, it is not 100% effective as there are other methods that can be used to determine the server type and version. It should be noted that for this to work, the Apache2 configuration variable ServerTokens should be changed from Prod (default) to Full so the line reads as follows:

1
 ServerTokens Full
SecAuditEngine

This directive allows more information about the methods of an attacker to be logged to the specified logfile. To turn this on to log every request object use the syntax:

1
 SecAuditEngine On

This is not very desirable as this produces a LOT of output. The more desirable version is the one used above:

1
 SecAuditEngine RelevantOnly

This logs only the interesting stuff that may be useful in back tracing the methods of an attacker.

SecAuditLog

This is the location of the audit log file. It is generally preferred to use absolute paths to files to ensure the correct path is being used.

SecFilterDebugLevel

This directive refers to the debug level logged to the specified logfile. The current value of 0 should be used on production systems. While the environment is in testing, a level of 1..4 should be used with increasing
verbosity between from 1 up to 4.

SecFilterDebugLog

This is the location of the audit log file. It is generally preferred to use absolute paths to files to ensure the correct path is being used.

We will add some lines to do some Selective filtering. Selective filters are used to handle some specific situations that cannot be targeted with site-wide policy. However you need to be careful of what you make site-wide policy since some of these security measures can break your current setup.

There are even more in depth uses where you can number rules and apply them to certain sets of directives and not to others. mod_security allows for very granular control. The in depth discussions on using these is beyond the scope of this document.

General

Since mod_security is a keyword driven engine, it will take the specified action on simple keyword matches. This is to say that anything that follows the directive SecFilter will engage the appropriate action. For example:

1
 SecFilter "&gt;applet"

If the <applet> tag appears anywhere in the request, then the log action specified above is taken.

XSS Attacks

To try to prevent some types of cross site scripting attacks, you can add the following lines to your configuration file:

1
2
 SecFilter "&lt;script"
 SecFilter "&lt;.+&gt;"

This tries to prevent Javascript injections or HTML injections.

Directory Traversal

Rarely will it be necessary for a user to traverse directories using the “../” construct. In order to prevent that, we can add the line:

1
 SecFilter "\.\./"
GET/HEAD Requests

With the use of these lines, we will not accept GET or HEAD requests that have bodies:

1
2
 SecFilterSelective REQUEST_METHOD "^(GET|HEAD)$" chain
 SecFilterSelective HTTP_Content-Length "!^$"
Unknown Requests

There are occasionally requests that come across (usually malicious) that we don’t know how to handle. At that point we let mod_security handle the request by adding the following lines:

1
 SecFilterSelective HTTP_Transfer-Encoding "!^$"

Conclusion

At this point, you should be capable of setting up a basic installation of mod_security. They are many more combinations of both simple and advanced techniques and directives that can be used to protect your server. By reading the documentation, you can have very granular control over your web server attack detection and prevention.

Originally Posted:

References

Mail::IMAPClient

Tuesday, January 23rd, 2007

Description: Recently, I have had the pleasure of getting knee deep into various aspects of Email. One of the things that I consistantly found myself wanting to do was to parse through it. I know the best way to do this is to connect to the IMAP server and download the messages. The best way I have come accross on how to accomplish this task is using Mail::IMAPClient.

CPAN: Mail::IMAPClient

Example 1:
Creating a connection to an IMAP server isn’t complicated. And once you are connected there are many things that you can do to manipulate messages. In this first example, I am merely going to show you how to connect to the mail server and download ALL the messages in specific folders:

# Always be safe
use strict;
use warnings;

# Use the module
use Mail::IMAPClient;

 $imap = Mail::IMAPClient->new( Server  => 'mail.server.com:143',
                                User    => 'me',
                              Password  => 'mypass')
        # module uses eval, so we use $@ instead of $!
        or die "IMAP Failure: $@";

 foreach my $box qw( HAM SPAM ) {
   # Which file are the messages going into
   my $file = "mail/$box";

   # Select the mailbox to get messages from
   $imap->select($box)
        or die "IMAP Select Error: $!";

   # Store each message as an array element
   my @msgs = $imap->search('ALL')
        or die "Couldn't get all messages\n";

   # Loop over the messages and store in file
   foreach my $msg (@msgs) {
     # Pipe msgs through 'formail' so they are stored properly
     open my $pipe, "| formail >> $file"
       or die("Formail Open Pipe Error: $!");

     # Send msg through file pipe
     $imap->message_to_file($pipe, $msg);

     # Close the messgae pipe
     close $pipe
       or die("Formail Close Pipe Error: $!");
   }

   # Close the folder
   $imap->close($box);
 }

 # We're all done with IMAP here
 $imap->logout();

Example 2:
In this next example, we are going to take advantage of some other methods that are provided by this useful little module. We will begin by using the same base as is in Example 1, but we will add some nuances in the middle for functionality.

Note: Messages don’t get immediately deleted with IMAP, only marked for deletion. They aren’t actually deleted until the box is expunged. In this case, it gets done after the looping over each mailbox is complete. This is to say that if the program gets interrupted in the middle that the messages won’t be deleted until the mailbox is officially issued an expunge command.

# Always be safe
use strict;
use warnings;

# Use the module
use Mail::IMAPClient;

 $imap = Mail::IMAPClient->new( Server  => 'mail.server.com:143',
                                User    => 'me',
                              Password  => 'mypass')
        # module uses eval, so we use $@ instead of $!
        or die "IMAP Failure: $@";

 foreach my $box qw( HAM SPAM ) {
   # How many msgs are we going to process
   print "There are ". $imap->message_count($box).
          " messages in the $box folder.\n";

   # Which file are the messages going into
   my $file = "mail/$box";

   # Select the mailbox to get messages from
   $imap->select($box)
        or die "IMAP Select Error: $!";

   # Store each message as an array element
   my @msgs = $imap->search('ALL')
        or die "Couldn't get all messages\n";

   # Loop over the messages and store in file
   foreach my $msg (@msgs) {
     # Pipe msgs through 'formail' so they are stored properly
     open my $pipe, "| formail >> $file"
       or die("Formail Open Pipe Error: $!");

     # Skip the msg if its over 100k
     if ($imap->size($msg) > 100000) {
       $imap->delete_message($msg);
       next;
     }

     # Send msg through file pipe
     $imap->message_to_file($pipe, $msg);

     # Close the messgae pipe
     close $pipe
       or die("Formail Close Pipe Error: $!");

     # Delete each message after downloading
     $imap->delete_message($msg);

     # If we are just testing and want to leave
     #  leave the messages untouched, then we
     #  can use the following line of code
     # $imap->deny_seeing($msg);
   }

   # Expunge and close the folder
   $imap->expunge($box);
   $imap->close($box);
 }

 # We're all done with IMAP here
 $imap->logout();

Sys::Hostname

Friday, January 19th, 2007

Description: Sys::Hostname is a relatively small, but very useful module. Just as the module name describes, it gets your system’s hostname. To paraphrase the module’s POD documentation, it will try every conceivable way to get the hostname of the current machine.

CPAN: Sys::Hostname

Example:
The one and only use for this module.

# Always be safe
use strict;
use warnings;

# Use the module
use Sys::Hostname;

# Get the hostname
my $host = Sys::Hostname::hostname();

# The above line can also be written as
#my $host = hostname;

print "You are on: $host\\n";

HTML::Entities

Thursday, January 18th, 2007

Description: When taking user input through any number of forms, there could be characters that you aren’t expecting. This is exactly what HTML::Entities was designed to handle. When getting the user input, it converts it into a form that can help in mitigating certain types of web based scripting attacks.

CPAN: HTML::Entities

Example 1:
The general example of using

1
encode_entities()

is probably also the most common. It basically says to encode everything in the string that its possible to encode.

# Always be safe and smart
use strict;
use warnings;

# Use the module
use HTML::Entities;

 my $html = "bad stuff here&#$%";
 $html = encode_entities($html);
 print "HTML: $html\\n";

__OUTPUT__
HTML: bad stuff here&#0

Example 2:
This is the slightly more specific example as it uses only specific sets of characters as the “unsafe” characters.

# Always be safe and smart
use strict;
use warnings;

# Use the module
use HTML::Entities;

 my $html = "bad stuff here&#$%";
 $html = encode_entities($html, "\\x80-\\xff");
 print "HTML: $html\\n";

__OUTPUT__
HTML: bad stuff here&amp;amp;#0

Example 3:
This is an example of

1
decode_entities()

which does the reverse. It checks the string to see if there are any HTML encoded characters and decodes them into their Unicode equivalent. This is the general version of

1
decode_entities()

which is similar to the version of

1
encode_entities()

demonstrated in Example 1.

# Always be safe and smart
use strict;
use warnings;

# Use the module
use HTML::Entities;

 my $html = "encoded: bad stuff here&amp;#0";
 $html = decode_entities($html);
 print "Unicode: $html\\n";

__OUTPUT__
Unicode: encoded: bad stuff here&#0

Mail::Sender

Wednesday, January 17th, 2007

Description: This is probably one of the modules that I use most frequently. I commonly write reporting and statistic generating scripts. When the data is finished being crunched, I then dump it into a scalar and send it off in an email. This is the module that does my dirty work for me.

CPAN: Mail::Sender

Example 1:
This is probably the most common implementation of Mail::Sender that I use. I generally build up what is going to go in the $EMAIL variable throughout the script and then eval it to ensure it gets put into the message.

# Always be safe
use strict;
use warnings;

# Use the module
use Mail::Sender;

# Remove Mail::Sender logo feces from mail header
$Mail::Sender::NO_X_MAILER = 1;

my $EMAIL = "This is the body of the email here.\\n";

  eval {
   # Create the anonymous Mail::Sender object
   (new Mail::Sender)->MailMsg(
   {
     smtp => "mail.mydomain.com",
     from => "\\"Postmaster\\" <postmaster\\@mydomain.com>",
     to   => "\\"Eric Lubow\\" <me\\@mydomain.com.com>",
     cc   => "\\"Tech Support\\" <tech\\@mydomain.com>",
     subject => "Test Email",
     msg  => "$EMAIL"
   })
     # Show the Mail::Sender error since $! isn't set here
     or die "Error Sending Mail: $Mail::Sender::Error";
  };

Example 2:
This example sends a mixed multipart message, with text in HTML and alternative text in plaintext. If the receiving user’s email client can read HTML messages, then the HTML portion of the message will be shown. Otherwise, the plaintext (alternative) will show. The iterative loop sends the email message to as many users as are in the array. The script just provides a qw (quote words) statement to be used in the loop. Your best bet would be to create an array of email addresses and iterate over the array:

 my @emails = ["eric@mydomain.com","techs@mydomain.com"];
 for my $rcpt (@emails) {
   ...
 }

Note: In this example, I have also included a subroutine to create a message boundary. I don’t particularly have anything major against the one’s that are generated by default, I just think they give away a little too much information for my liking. Mine is slightly more random.

# Always be safe
use strict;
use warnings;

# Use the module
use Mail::Sender;

# Remove Mail::Sender logo feces from mail header
$Mail::Sender::NO_X_MAILER = 1;

 for my $rcpt (qw( me@mydomain.com tech@mydomain.com ) ) {
   my $sender = new Mail::Sender
    { smtp => '127.0.0.1',
       from => "$from_name <postmaster\\@mydomain.com>",
       boundary => make_boundary()
    }
    or die "Error in mailing: $Mail::Sender::Error\\n";

   # Begin the message, tells the header what message type
   $sender->OpenMultipart(
     { to        => "eric\\@mydomain.com",
        subject   => "Test Email",
        headers   => "Return-Path: postmaster\\@mydomain.com\\r\\n",
        multipart => "mixed",
     })
     or die "Can't Open message: $sender->{'error_msg'}\n";

     # Change the content-type
     $sender->Part( { ctype => 'multipart/alternative' });

     # Put the plain text into the email (aka the alternative text)
     $sender->Part(
       { ctype       => 'text/plain',
         disposition => 'NONE',
         msg          => "This is a test message.\n\nHere is the plaintext.\\n"
       });

     # Put the HTML text into the HTML (what SHOULD be displayed)
     $sender->Part(
       { ctype       => 'text/html',
         disposition => 'NONE',
         msg         => "This is a <B>TEST</B> message.\\n<br>\\n"
       });

     # The message isn't sent until it's Close()'d
     $sender->Close()
       or die "Failed to send message: $sender->{'error_msg'}\\n";
 }

#######################################################
# Function: Creates a random message boundary
# Parameters: None
# Return: Boundary
# Notes: Bounaries are 43 characters in length
#######################################################
sub make_boundary {
  my ($boundary, $char, $wanted) = ('','',43);

  open R, '<', "/dev/urandom";
  while ($wanted) {
    read (R, $char, 1)
      or die "Couldn't read from /dev/urandom: $!\n";
    if ($char =~ /[A-Za-z0-9]/) {
      $boundary .= $char;
      --$wanted;
    }
  }
  close(R);

  return $boundary;
}

IO::Socket::INET

Tuesday, January 16th, 2007

Description: When I need to interact with the raw IO of telnetting to a port or creating a hand rolled implementation of my own service, I use IO::Socket. My most recent endeavor was for a need to check weather or not services were running on a regular basis. I will show some excepts from the code later on in this post.

Although IO::Socket::INET is just a frontend for a good number of Perl’s builtin functions, I find it handy as it is generally a little more sane.

CPAN: IO::Socket::INET

Note: IO::Socket::INET is a subclass of IO::Socket.

Example 1:
This example is the socket portion of a script that checks to see if certain daemons are alive. It does this by connecting to the port that they should be listening on. If it receives a response, then the daemon is alive, if not, it restarts the daemon and tries contacting it again. As you can see below, the response itself doesn’t matter, just that there is a response being received.

# Always be safe and smart
use strict;
use warnings;

# Use the module
use IO::Socket;

 # Prototype the socket
 my $socket = new IO::Socket::INET(
                   PeerAddr => "localhost",   # Hostname/IP
                   PeerPort => "smtp(25)",   # Service (Port)
                   Proto    => "tcp",            # Protocol
                   )
   or die "Socket bind error: $!";

 # Create the socket
 $socket
   or die "Socket error: $!";

 # Close the socket
 close $socket
   or die "Socket close error: $!";

Example 2:
On the other side of the fence, we can create a listening socket that can act as a server. If you run this little script and then connect to the specified port it will say something along the following:

1
Welcome 127.0.0.1 to a simple socket server.

The server script will listen indefinitely (or at least until you press CTRL-C to make it exit). To test out the script and ensure you receive something similar to the above response (with your IP instead of 127.0.0.1), from the linux command prompt, type the following (replace with your IP and port):

1
$ telnet localhost 12345
# Always be safe and smart
use strict;
use warnings;

# Use the module
use IO::Socket;

 # Listen port
 my $port = 12345;

 # Prototype the socket
 my $server = new IO::Socket::INET(
   Listen   => SOMAXCONN, # Max connections allowed by system
   Reuse    => 1,               # Reuse connections
   LocalAddr => "localhost", # Listen hostname
   LocalPort => $port,        # Listen port
   Proto    => "tcp",           # Listen protocol
 ) or die "Socket bind error: $!";

 # Listen for the client until we get a CTRL-C
 while (my $client = $server->accept()) {
   # Force the data out of the stack
   $client->autoflush(1);

   # Say hello to my little client
   print $client "Welcome ". $client->peerhost ." ".
                 "to a simple socket server.\n";
 }

 # Close the server when we are done
 close $server
   or die "Socket close error: $!";

File::Pid

Thursday, January 11th, 2007

Description: Consistantly writing programs that either act as daemons or take a long time to run depending on the input, I find it necessary to track weather or not they are still running. Lots of programs do this and its generally called state or status tracking. One of the most common ways to do this is to use a pid file.

The most common method is to create a file with the naming scheme program_name.pid. The contents of the file is the pid of the program. Then at the conclusion of the program, the file is removed. The pid file is generally stored in the OS’s state directory (in linux). Generally being a Debian or Ubuntu, the state information is stored in the /var/run/ directory. I also frequently use EnGarde Secure Linux which also has a state directory of /var/run/.

CPAN: File::Pid

Note: In the spirit of TIMTOWTDI, the process itself isn’t overly complex. To illustrate that, I will show one method of accomplishing this goal without using a Perl module. I will also be using the Perl special variable $$. To quote the Perl Special Variables page:

  • $PROCESS_ID
  • $PID
  • $$

    The process number of the Perl running this script. You should consider this variable read-only, although it will be altered across fork() calls. (Mnemonic: same as shells.)

    Note for Linux users: on Linux, the C functions getpid and getppid() return different values from different threads. In order to be portable, this behavior is not reflected by $$, whose value remains consistent across threads. If you want to call the underlying getpid, you may use the CPAN module Linux::Pid.

  • As you will likely note, although potentially less portable across OS’s, using File::Pid is much cleaner and easier.

    # Declare your variables
    my $pid_file = "/var/run/myprog.pid";
    local *PIDFILE;
    
    # Check for the PID file's existance
    if (-e $pid_file) {
      die ("Program already running or ended prematurely\\n");
    }
    
    # Create the PID file
    # Use the 3 argument form of open for safety
    open PIDFILE, ">", $pid_file;
    print PIDFILE "$$\\n";
    close PIDFILE;
    
    # XXX - Program here
    
    # Remove PID file
    unlink $pid_file
      or die "Error unlinking $pid_file: $!";
    

    Example 1:
    Using the basic example of just creating the PID file, checking if the program is already running, and removing the PID file, let’s take a look at the OO (Object Oriented) example:

    # Always be safe & strict
    use strict;
    use warnings;
    # Use the module
    use File::Pid;
    
     # Create the PID object
     # Ensure you put a name that won't clobber
     #   another program's PID file
     my $PID = File::Pid->new({
       file  => '/var/run/myprog.pid',
     });
    
     # Write the PID file
     $PID->write;
    
     # Check for the previous instance
     #  and die if PID file already exists
     if (my $oldPID = $PID->running()) {
       die "myprog already running: $oldPID\n";
     }
    
     # XXX - Program here
    
     # Remove the PID file
     $PID->remove;
    

    File::Find

    Wednesday, January 10th, 2007

    Description: File::Find is a pretty straightforward and useful module. I often find myself needing to hunt down a bunch of files or parse through a particular grouping of files. Because of TIMTOWTDI, I usually choose to use File::Find based on its ease of use.

    CPAN: File::Find

    Example 1:
    Being a System’s Administrator, I am usually hunting for something in a logfile. Therefore, I like to parse through all the logs for a particular program or daemon that I can get my hands on.

    # Always use these
    use strict;
    use warnings;
    # Use the module itself
    use File::Find;
    
    # Declare your variables
    my @files;
    my $dir = "/var/log/";
    
     # Actually find the file
     #  -f tests to see if it is a file (not a device or symlink, etc)
     #  Matches the RE for "mail.log.*"
     #    in the directory $dir
     find(
       sub { push @files, $File::Find::name if -f && /mail\.log.*/ },
       $dir);
    
     # Iterate over the files and print those found
     for my $file (@files) {
       print "File: $file\\n";
     }
    

    Email::Find

    Tuesday, January 9th, 2007

    Description: Email::Find is a module for finding a subset of RFC 822 email addresses in arbitrary text. The addresses it finds are not guaranteed to exist or even actually be email addresses at all, but they will be valid RFC 822 syntax. Email::Find will perform some heuristics to avoid some of the more obvious red herrings and false addresses, but there’s only so much which can be done without a human. (Note: Taken from the author’s description)

    CPAN: Email::Find

    Example 1:
    I often find myself with a list of email address in an Excel spreadsheet or on an email that someone sent to me with a list of names. Normally I have to take this list and resend a message to the recipients without allowing all the addresses to be seen. Since according to RFC 2822, there are multiple implementations of how a message with BCC recipients can be sent, I find it is easier to break the recipients down myself and have them presented to me in a way that I can use them in a quantity in which they are useful to me.
    Excerpt from RFC 2822:

    The “Bcc:” field (where the “Bcc” means “Blind Carbon Copy”) contains addresses of recipients of the message whose addresses are not to be revealed to other recipients of the message. There are three ways in which the “Bcc:” field is used. In the first case, when a message containing a “Bcc:” field is prepared to be sent, the “Bcc:” line is removed even though all of the recipients (including those specified in the “Bcc:” field) are sent a copy of the message. In the second case, recipients specified in the “To:” and “Cc:” lines each are sent a copy of the message with the “Bcc:” line removed as above, but the recipients on the “Bcc:” line get a separate copy of the message containing a “Bcc:” line. (When there are multiple recipient addresses in the “Bcc:” field, some implementations actually send a separate copy of the message to each recipient with a “Bcc:” containing only the address of that particular recipient.) Finally, since a “Bcc:” field may contain no addresses, a “Bcc:” field can be sent without any addresses indicating to the recipients that blind copies were sent to someone. Which method to use with “Bcc:” fields is implementation dependent, but refer to the “Security Considerations” section of this document for a discussion of each.

    To accomplish this, I use variations of the following simple script:

    # Always use these
    use strict;
    use warnings;
    # Use the module itself
    use Email::Find;
    
    # Declare your variables before using them
    my (@emails, $file, $counter);
    # Limit the number of email addresses per line
    my $max = 20;
    
    # Code block to slurp the file on $ARGV[0]
    {
      # Slurp the file by changing the Perl special variable
      local $/ = undef;
      local *FILE;
      open FILE, '<', "$ARGV[0]"
        or die "File Open Error: $!";
      $file  = <FILE>;
      close FILE
        or die "File Close Error: $!";
    }
    
      # Prototype the pushing of all the addresses
      #  found onto an array leaving the original text
      my $finder = Email::Find->new(sub {
            my ($email, $orig_email) = @_;
            push @emails, $email->format;
            return $orig_email;
            });
      # Actually push the addresses found in $file
      $finder->find(\$file);
    
      # Loop and print $max email addresses per line
      #  separated by a semi-colon ';' and then each
      #  line separated by two carriage returns '\n'
      for my $address (@emails) {
        $counter++;
        print "$address;";
        print "\n\n"
          if $counter == $max;
        $counter = 0
          if $counter == $max;
      }
    

    Perl Modules Introduction

    Monday, January 8th, 2007

    Before I go into any modules, I think I should explain what I am doing and why I am doing it. And although I am sure that this is done elsewhere and I am not a fan of reinventing the wheel, I am a fan of making my own life easier and hopefully other’s lives as well.

    I often find myself using / reusing the same modules for small scripts. Either they are Perl one-liners or very short scripts to accomplish a semi-common task. Rather than going through and figuring out how I used the module so effectively in the previous instance, I have resorted to storing examples of how I implement various modules in my local CVS repository. Although the whole versioning and tagging system allows me the flexibility of using more than 1 example per module, it is also somewhat restrictive in its ability to allow for search (unless you checkout the whole repository and grep or other overkill or tedious methods).

    Then I decided that a good way to do this would be to use a blog. This is the method I chose for a few reasons.

    1. Searchable
    2. I can easily search for the module I want to use taking advantage of either Google‘s index of my site (or another search engine or blog search engine). I can also easily search it using the internal searching methods of the blog.

    3. Central Location
    4. As I said before…although I am sure that there are other places that do this (including the POD source of many modules and CPAN), I want a central location with easy to understand code.

    5. Everyone Else
    6. There are many other’s out there who sometimes don’t understand the code in the POD or in some other code examples. Even larger code repositories and search engines like Krugle are still a little kludgey for basic examples. This also allows for people to contribute if they so desire.

    And another great thing about this is that if I get a lot of examples in here, then search engine sites and code repositories will index it and it will be useful to more than just a select few elite who read this blog.

    The general rule of this for me is to explain a little about the module, give an example or two from real life, and provide some useful comments to help explain what the code is doing. Since I don’t plan on re-inventing the wheel (as stated more than a few times), I may just copy and paste from the module author (giving credit where credit is due of course). I encourage anyone with simple enough code to provide examples at any point as well. Enjoy.