Posts Tagged ‘ bash ’

Using Vi Mode Everywhere

Tuesday, March 15th, 2011

Not literally everywhere, but more places than usual. I have been looking for this solution for a long time and finally found it. Anyone who has ever worked around me knows that I do basically everything in Vi.

Counting Frequencies of Frequencies

Monday, August 16th, 2010

Lots of people forget about the usefulness of the core utilities (the tools available in Bash). I am even pretty guilty of it at times with such quick and easy things like Perl, Ruby, or Python that allow you to process items from the command line. However, they load up an entire interpreter. It is usually better to use the coreutils.

Multiple Input Locations From Bash Into Ruby

Thursday, May 13th, 2010

I have been trying to figure out how, while using OptionParser to be able to check for files being input on the command line and if they don’t exist, check other input streams (like Bash). This initially wasn’t very easy since input streams are blocking. So with a little help from friends (thanks roberto), I was able to use his method of non-blocking IO and wrap it in a begin/rescue block. I also took a little advice given in this Stack Overflow question called Best Practices with STDIN in Ruby.

Count Instead of Sequence

Friday, March 26th, 2010

I use Bash one liners a lot. I think they are an important part of any programmers and sysadmins toolkit. If you can’t write a bash one liner, even a simple iterator, then you really need to learn. I promise it will make your life infinitely more pleasant.

Frequently I find myself writing things that require a loop or an increment of numbers. A good example would be like something that would walk over my web servers and check their uptime, load averages, etc. Using seq, that’s easy. But since Mac OS X doesn’t come with the seq command, I would previously improvise.

Git Branch Name in Your Bash Prompt

Friday, December 11th, 2009

I work with a few repositories at any given time. And during that time, I typically have multiple branches created for each repository. I figured that it would make my life easier if I knew which branch and/or repository I was working in. Luckily, very little hackery is required here since the git distribution already comes with such a tool. (Note: If you didn’t build Git from source, then you may not have this file.)

Printing New Lines in Bash

Thursday, October 22nd, 2009

Ran across this the other day and decided it required sharing. If you want to print a new line ‘\n‘ in an echo statement in bash, one would think its just as simple as:

beacon:~ elubow$ echo "This is a test\n"
This is a test\n

The problem is that this doesn’t interpolate the newline character. (For more information on interpolation, see Wikipedia here.) In order to have the newline interpolated, you need to add the command line switch ‘-e‘.

beacon:~ elubow$ echo -e "This is a test\n"
This is a test

This will force Bash to interpolate any non-literal characters in the quotes. Note: Unlike Perl, single or double quotes don’t matter here when Bash is deciding whether or not to interpolate the new line characters.

Quick Date Calculations With Date

Tuesday, July 21st, 2009

I frequently find myself needing to do quick date calculations in order to make scripts run when I want them to or how I want them to. Usually Date::Calc is just a bit too heavily, especially if it’s something as simple as a BASH script. As it happens, date is quite a powerful tool for some command line fu.

For example, to find the first and last day of last month:

FIRST_DOLM=`date -d "-1 month -$(($(date +%d)-1)) days" "+%Y-%m-%d"`
LAST_DOLM=`date -d "-$(date +%d) days" "+%Y-%m-%d"`

You can do the same thing using the date command on a mac with a slightly different set of switches:

FIRST_DOLM=`/bin/date -v1d -v-1m "+%Y-%m-%d"`
LAST_DOLM=`/bin/date -v31d -v-1m "+%Y-%m-%d"`

Both of these will produce the same results:


Cleaning Up Long Conditionals With Grep

Tuesday, April 7th, 2009

Every so often I am faced with testing a few conditionals before dropping into another control structure. If you have to test out a few conditionals, then its likely a dispatch table won’t be useful. If you have a lot of conditionals to test, you’ll likely not want to deal with an ugly expression like the following:

if ( (defined $SITE{$partner}{'foo'} && ($SITE{$partner}{'foo'} > 0) ) and
        ( (defined $assoc{'foo'} && ($assoc{'foo'} > 0)) or
          (defined $assoc{'bar'} && ($assoc{'bar'} > 0)))
   ) {
print "We're here!\n";

One of the ways to deal with it to to use grep. Since grep returns the number of elements in the array that evaluate to true (when called in a scalar context), I can do the following to make it work:

my @foo = [ $SITE{$partner}{'foo'}, $assoc{'foo'}, $assoc{'bar'} ];
if ( (grep {defined($_) and $_ > 0} @foo) > 2) { print "We're here!\n"; }

The reason this works is that when called in the scalar context, grep only returns the number of elements that evaluate to true. Since there are 3 elements in the array, the return value is greater than 2, then the entire expression evaluates to true. This is not only a lot easier to read and a lot cleaner to write, but it makes for easy additions to the conditional testing if necessary. Although changing the context in which functions are called is common, it is easily forgotten. Grep called in scalar context can be easily manipulated (as above) to add readability to your program.

Deleting Lots Of Files (SysAdmin problem solving exercise)

Monday, December 17th, 2007

Since I know I am not the first (or the last) to make a typo in logrotate and not catch it for a while…someone else must have been in the position of having to delete a lot of files in the same manner. I recently learned that, as usual, there is more than one way to handle it.

To put the situation in context, I basically allowed thousands of mail.* files to be created. These files littered the /var/log/ directory and basically slowed down the entire file system access. I figured out this a number of ways.

The first way was when I tried to do an ls anywhere, it would just hang. My first reaction was to check to see what was eating up the CPU. To do this, I did a top. I noticed that logrotate was hogging all the CPU cycles. Since I know that logrotate basically only operates on one parent directory (by default) /var/log, I headed on over there and did an ls. Once again, it just hung. Then I figured the file system was slow and decided to check out some file system information. The next two commands I ran were df -h and df -i. I ran the df -h to see if we were out of disk space (and yes I lazily use human readable format). I ran the second to check to see how many inodes were in use. (For more information on inodes, check out the wikipedia entry here).

Now that I know the system is short on inodes, I checked out the output of lsof. Now I know that we have some serious problems in the /var/log dir. After some quick investigation, I realized that there were too many mail.* files. How do I get rid of them? Glad you asked… Let’s assume that we want to delete ALL the mail.* files in the /var/log directory.

1) The easiest way is to do it with find:
1a) Using find‘s delete command:

[root@eric] /var/log # find ./ -type f -name "mail.*" -delete

1b) using find‘s exec command with rm:

[root@eric] /var/log # find ./ -type f -name "mail.*" -exec rm -rf '{}' \;

These will work, but either will be slow since they doesn’t do batch execution.

2) A slightly more preferred way is to use bash:

[root@eric] /var/log # for n in mail.*; do rm -v $n; done;

This is a little faster, but will still be relatively slow since there is no batch execution. (Note: The -v in the rm will cause quite a bit of output since it is showing you EVERY file it deletes. Feel free to leave this out if you really screwed up.)

3) The actual preferred method is to use find:

[root@eric] /var/log # find ./ -type f -name "mail.*" | xargs rm -f

I believe this is the preferred method because although it removes the files one at a time, it is more efficient for the file system since it batches it up.

There are certainly other ways to accomplish this task. It can always be done with a Perl one-liner or even using some Perl modules to save some time. These are just a few ideas to point someone in the right direction.