Go to content Go to navigation Go to search

File::Bidirectional

March 9th, 2007 by eric

Description: The author of this module notes that it is best used, especially by him, when reading or manipulating log files. I have a tendency to use it for the exact same thing, especially when looking for context around captured lines.

CPAN: File::Bidirectional

Note:
Although I would like to note that using the tie’d interface as I have done takes approximately 2 1/2 times as long as a regular file read according to benchmarks, it is still a very handy tool and allows one not to reinvent the wheel.

Example 1:
Here we are going to go through a log file and when we hit the time stamp we want, we are going to change directions and go back through. There is no real reason to change direction here, I am merely demonstrating how it would be accomplished.

# Always use these
use strict;
use warnings;

# Use the module itself
use File::Bidirectional;

# Define the log file to be read
my $log = "/var/log/log_file";

# Open the logfile by tie'ing it to the module
#  This is exactly the same as File::ReadBackwards
tie *LOG, "File::Bidirectional", "$log", {mode => 'backward'}
   or die ("$log tie error: $!");

# Iterate over the logfile
while (my $line = ) {

  # Split the log line
  my @entry = split(/\s+/, $line);

  # Take the timestamp and check if we
  #   have hit our threshold yet
  # Get the line # then change direction
  if (time_reached($entry[0]) == 1) {
    $line_num = (tied *LOG)->line_num();
    (tied *LOG)->switch();
  }
}

# Cleanup
untie (*LOG);

Ensuring Proper New DST Compliance

March 6th, 2007 by eric

By now, if you haven’t heard about the change in daylight savings time (DST), then you need to break out from under your rock and update your servers. That’s where luckily, I come in. No, I won’t update your servers for you (unless you pay me and even then it’s iffy), but I will guide you along in updating them.

Before I jump headlong into helping you out, I feel as though (since it’s my blog anyway) that I just want to rant for a minute…

<rant>I heard that we tried this a while back and they had kids walking to school in the dark…which in and of itself isn’t good. So we figure, let’s try it again. If we can just save a little bit of money by everyone using 1 hour or so less power for 2 weeks, that everything would be dandy. I don’t think anyone took into account the man hours that it would take to update all the software and all the hardware and every little thing that uses dates and has relied on the system that we used for many years. And amusingly enough, if someone doesn’t get everything perfect with the updates for the power systems, then who knows, this may require people to use more power to make up for mistakes. Who’d have thought something like this has the possibility to backfire?</rant>

The first thing you need to do is to check to see if you are updated and ready to go. To do that, if you are in the EST/EDT timezone and your system isn’t updated, then it will look like this when you give it the following command:
# zdump -v /etc/localtime | grep 2007
/etc/localtime Sun Apr 1 06:59:59 2007 UTC = Sun Apr 1 01:59:59 2007 EST isdst=0 gmtoff=-18000
/etc/localtime Sun Apr 1 07:00:00 2007 UTC = Sun Apr 1 03:00:00 2007 EDT isdst=1 gmtoff=-14400
/etc/localtime Sun Oct 28 05:59:59 2007 UTC = Sun Oct 28 01:59:59 2007 EDT isdst=1 gmtoff=-14400
/etc/localtime Sun Oct 28 06:00:00 2007 UTC = Sun Oct 28 01:00:00 2007 EST isdst=0 gmtoff=-18000

This is still on the old timezone setting since it says April 1. If that’s too confusing for you, here’s a handy dandy little one-liner that maybe a little more straightforward:
# zdump -v /etc/localtime | grep 2007 | grep -q "Mar 11" && echo "New DST Compliant" || echo "Not New DST Compliant"
Not New DST Compliant

It will say “Not New DST Compliant” if your setup isn’t currently in compliance with the new DST setup.

To become DST compliant, all you really have to do is update your versions libc6 and locales. This update will ensure that your timezone data is up to date and your system will then respond to the time change accordingly. On a Debian, Ubuntu, or other apt based system, you will only have to do the following (note the ‘#’ means we are already root, if you are on Ubuntu, put a sudo in front of the command):
# apt-get update && apt-get install libc6 locales
Gentoo:
# emerge --sync && emerge sys-libs/timezone-data

If all went well, then you should be able to execute the same command as above and it will look like this:
# zdump -v /etc/localtime | grep 2007
/etc/localtime Sun Mar 11 06:59:59 2007 UTC = Sun Mar 11 01:59:59 2007 EST isdst=0 gmtoff=-18000
/etc/localtime Sun Mar 11 07:00:00 2007 UTC = Sun Mar 11 03:00:00 2007 EDT isdst=1 gmtoff=-14400
/etc/localtime Sun Nov 4 05:59:59 2007 UTC = Sun Nov 4 01:59:59 2007 EDT isdst=1 gmtoff=-14400
/etc/localtime Sun Nov 4 06:00:00 2007 UTC = Sun Nov 4 01:00:00 2007 EST isdst=0 gmtoff=-18000

And if you are still being lazy, here is the shorthand command again and the output that you should get:
# zdump -v /etc/localtime | grep 2007 | grep -q "Mar 11" && echo "New DST Compliant" || echo "Not New DST Compliant"
New DST Compliant

That should be all you have to do. Remember that if you aren’t in MY timezone (Eastern Time, USA), then your output will be a little different.

Update: The error in the Gentoo update has been fixed. Thanks to Neo for pointing it out.

Underused Tools

March 5th, 2007 by eric

There are a lot of tools for administration and networking that generally go unused. They are very helpful in both diagnostics and general administration. There are even some tools that come installed with linux and go unused and unheard of. Here I am going to cover a mere few of my favorite and hope that they work for you as well.

  1. traceproto
    The first tool I want to cover is one of my favorite tools when writing firewall scripts and is a close relative of traceroute; it’s called traceproto. traceproto doesn’t come installed by default on most linux systems. It is a replacement (or even just a complement) for traceroute that goes the extra mile. Like traceroute, you can change ports and ttl (time to live) on your queries. But the extra mile appears where you can specify whether to use tcp, udp, or icmp when you specify the ports. You can also specify the source port of the network traffic.
    The way that I make best use this tool is when I am writing firewall scripts. For instance, when I allow ntp through on a firewall, it can sometimes be difficult to test if my firewall rules are letting the packets through (since I have multiple levels of firewalls). Therefore, I use traceproto as follows (ntp is on udp port 123):
    root@tivo:~# traceproto -d 53 -p udp ns1.myserver.com
    traceproto: trace to ns1.myserver.com (1.2.3.4), port 53
    ttl 1: ICMP Time Exceeded from 192.168.1.1 (192.168.1.1)
    0.83300 ms 0.67900 ms 0.71300 ms
    ttl 2: ICMP Time Exceeded from 10.75.128.1 (10.75.128.1)
    11.577 ms 6.1550 ms 6.4960 ms
    ... Removed for brevity ...
    ttl 11:no response no response no response
    ttl 12: UDP from myserver.com (1.2.3.4)
    132.07 ms 126.28 ms 125.88 ms

    hop : min / ave / max : # packets : # lost
    -------------------------------------------------------
    1 : 0.67900 / 0.74167 / 0.83300 : 3 packets : 0 lost
    2 : 6.1550 / 8.0760 / 11.577 : 3 packets : 0 lost
    3 : 5.9680 / 7.0697 / 7.6650 : 3 packets : 0 lost
    4 : 0.0000 / 0.0000 / 0.0000 : 0 packets : 3 lost
    5 : 0.0000 / 0.0000 / 0.0000 : 0 packets : 3 lost
    6 : 8.8930 / 12.198 / 15.810 : 3 packets : 0 lost
    7 : 0.0000 / 0.0000 / 0.0000 : 0 packets : 3 lost
    8 : 9.2340 / 24.556 / 32.438 : 3 packets : 0 lost
    9 : 9.8230 / 13.669 / 18.890 : 3 packets : 0 lost
    10 : 0.0000 / 0.0000 / 0.0000 : 0 packets : 3 lost
    11 : 0.0000 / 0.0000 / 0.0000 : 0 packets : 3 lost
    12 : 125.88 / 128.08 / 132.07 : 3 packets : 0 lost
    ------------------------Total--------------------------
    total 125.88 / 22.834 / 132.07 : 21 packets : 15 lost

  2. pstree, pgrep, pidof
    Although these are 3 separate tools, they are all very handy for process discovery in their own right.

    To take advantage of of the pidof command, you just need to figure out which program you want to know about its family (parent and children). 2 ways to demonstrate this would be to use either kthread or apache2 as follows:
    # pidof apache2
    29297 29291 29290 29289 29245 29223 29222 29221 20441
    # pidof kthread
    6

    By typing pstree, you will see exactly what it is capable of. pstree outputs an ASCII graphic of the process list by separating it into parents and children. By adding the -u option to pstree, you can see if your daemons made their uid transitions. This is also an extremely useful program for displaying SELinux context of each process (by using the -Z option if pstree was built with it). To see the children of kthread which we found above was pid 6, we can use these commands in conjunction.
    # pstree `pidof kthread`
    kthread-+-aio/0
    |-kacpid
    |-kblockd/0
    |-kgameportd
    |-khubd
    |-kmirrord
    |-kpsmoused
    |-kseriod
    `-2*[pdflush]

    And finally pgrep. There are many ways to make use of pgrep. It can be used like pidof:
    # pgrep -l named
    18935 named

    We can also list all processes that are being run that aren’t being controlled by controlling port 1 (pts/1):
    # pgrep -l -t pts/1 -v
    1 init
    2 ksoftirqd/0
    3 watchdog/0
    4 events/0
    ... Removed for brevity ...
    10665 getty
    18975 named
    19009 qmgr
    25447 sshd
    25448 bash
    29221 apache2

  3. tee
    There are sometimes commands that can take a long time to run. You want to see the output, but you also want to save it for later. How can we do that. We can use the tee command. This sends the output to STDOUT and send (or append) to a filehandle. For simplicity, I will show you an example of tee using an df.
    df -h | tee -a snap_shot
  4. tac
    Everyone knows about cat, it’s what we use to list the entire contents of a file. cat has a little known cousin that is usually installed by default on a system called tac. It prints the entire contents of a file in reverse.
  5. fuser
    fuser displays the process id of all processes using the specified file or file system. This has many handy uses. If you are trying to unmount a partition and want to know why its still busy, then run fuser on the filesystem and find out which processes are still using the device. fuser is even nice enough to tell you what kind of files are using the files or file systems. For example, I want to umount /root/, but I can’t and I don’t know why:
    # fuser /root/
    /root: 29475c 29483c

    Hmm, c means that I am currently in the directory. Maybe I need to watch what I’m doing.

Most of these tools don’t fall into the same category, but they are all useful in their own right. I hope you can make as good use of them as I do. There are many more little known tools that come with many linux installs by default and this is a just a few of the common ones that I take advantage of on a regular basis.

Next Entries »