Erics Tech Blog » bash

Using Vi Mode Everywhere

eric — Tue, 15 Mar 2011 07:00:07 +0000

Not literally everywhere, but more places than usual. I have been looking for this solution for a long time and finally found it. Anyone who has ever worked around me knows that I do basically everything in Vi.

Not only do I use it to edit files, but I use it as an IDE for development (even on my Mac instead of Textmate). So the natural extension is for me to use it in the command prompt as well. So in my .bashrc file, I have the line:

set -o vi

This allows me to navigate the bash console with the usual vim suspects: h, j, k, l. In addition to that, I also get some fun ones like word movement w and the dw that goes along with it.

But the big winner for me is now I am able to use the vim environment and movement keys inside the irb (Ruby), Mongo, MySQL consoles (still not Redis though). To do that, just add the following lines to the following files:

$ echo "bind -v" >> ~/.editrc
$ echo "set editing-mode vi" >> ~/.inputrc

No related posts.

Counting Frequencies of Frequencies

eric — Mon, 16 Aug 2010 12:30:13 +0000

Lots of people forget about the usefulness of the core utilities (the tools available in Bash). I am even pretty guilty of it at times with such quick and easy things like Perl, Ruby, or Python that allow you to process items from the command line. However, they load up an entire interpreter. It is usually better to use the coreutils.

I’ll give you the specific example I had to deal with, but this can be extrapolated out and I’m sure reused for other purposes. I have lists of email addresses of who has received emails that were sent over the past 7 days. I do this with a find command since the address files are created nightly. They are laid out with 1 email address (and some other meta data) per line. You’ll notice the awk in the find command; that’s just to extract the email address from the line to make things easier to work with.

The overall goal here is find out how many times people were emailed over the past 7 days. Another way of saying it is, how many people received 1,2,3,4, etc mailings over the past week. This is an exercise in aggregation.

First, here is everything in its entirety and then I will proceed to go through it all piece by piece.

$ find /nfs/mailings/lists/ -maxdepth 3 -ctime -7 -name "master_list.txt" \
-exec awk '{print $2}' '{}' >> /tmp/emails.txt \; && cat /tmp/emails.txt | \
awk '{ for (i=1;i< =NF;i++) count[$i]++ } END { for (i in count) print count[i], i }' $* |\
awk '{print $1}' | awk '{ for (i=1;i<=NF;i++) count[$i]++ } \
END { for (i in count) print count[i], i }' $* | awk '{print $2,$1}' | \
sort -n && rm -rf /tmp/emails.txt
1 93278
2 21415
3 16924
4 3064131
5 548421
6 102

A quick breakdown goes as follows.

1) Find all mailing list master files (-name) that aren’t archived (-maxdepth) that were created in the last 7 days (-ctime) and print their id (exec awk) to a tmp file. Everything here needs to be concatenated so as not to clobber the emails added by the previous day.

find /nfs/mailings/lists/ -maxdepth 3 -ctime -7 -name "master_list.txt"\
-exec awk '{print $2}' '{}' >> /tmp/emails.txt \;

2) The next piece word frequency counts the emails and prints them to STDOUT in a manner. This is to say that the first column is a count of how many times the second column has received an email over the past 7 days. Now we know how many emails each individual has received and we need to aggregate again.

cat /tmp/emails.txt | awk '{ for (i=1;i< =NF;i++) count[$i]++ } \
END { for (i in count) print count[i], i }' $*

3) Because I am using pipes, instead of printing the previous step to STDOUT, I only print the first column (count) and then count the frequency of that.

awk '{print $1} | awk '{ for (i=1;i< =NF;i++) count[$i]++ } \
END { for (i in count) print count[i], i }' $*

4) Then I swap the columns so I get . This is much more logical and is the way we generally read tables.

awk '{print $2,$1}'

5) Finally I sort that result numerically ascending and remove the temp file so I don't cat things to it again.

sort -n && rm -rf /tmp/emails.txt

6) We get our output in an easy to read format (Note: I added the headers).

Freq Total
1 93278
2 21415
3 16924
4 3064131
5 548421
6 102

Multiple Input Locations From Bash Into Ruby

eric — Thu, 13 May 2010 12:00:59 +0000

I have been trying to figure out how, while using OptionParser to be able to check for files being input on the command line and if they don’t exist, check other input streams (like Bash). This initially wasn’t very easy since input streams are blocking. So with a little help from friends (thanks roberto), I was able to use his method of non-blocking IO and wrap it in a begin/rescue block. I also took a little advice given in this Stack Overflow question called Best Practices with STDIN in Ruby.

First we need to get the file list off the command line and assume that anything left are files.

@opts = OptionParser.new do |o|
...
end
@opts.parse!(args)
@files = args

Since we are using threads, I open up a thread for STDIN and killed it when we run out of input.

require 'fcntl'
threads = Array.new

# Set $stdin to be non-blocking
$stdin.fcntl(Fcntl::F_SETFL,Fcntl::O_NONBLOCK)

threads[0] = Thread.new {
begin
$stdin.each_line { |line| puts "STDIN: #{line}" }
rescue Errno::EAGAIN
threads.delete(0) # Remove this thread since we won't be reading from $stdin
end
}.run

Now its time for the files.

@files.each do |file|
threads.push Thread.new {
# do stuff with 'file'
}
end

# Put it all together and have the threads run
threads.each { |thread| thread.join }

Using these code snippets, you will be able to use input from both files on the command line and STDIN:

$ myscript.rb file1 file2
$ cat foo | myscript.rb file1 file2
$ myscript.rb file1 file2 < foo

Count Instead of Sequence

eric — Fri, 26 Mar 2010 10:30:29 +0000

I use Bash one liners a lot. I think they are an important part of any programmers and sysadmins toolkit. If you can’t write a bash one liner, even a simple iterator, then you really need to learn. I promise it will make your life infinitely more pleasant.

Frequently I find myself writing things that require a loop or an increment of numbers. A good example would be like something that would walk over my web servers and check their uptime, load averages, etc. Using seq, that’s easy. But since Mac OS X doesn’t come with the seq command, I would previously improvise.

# Linux system
$ for n in `seq 1 6`; do echo -n "web${n}: "; ssh web$n uptime; done;
web1: 18:36:56 up 82 days, 22:50, 4 users, load average: 0.03, 0.00, 0.00
web2: 18:36:56 up 82 days, 22:48, 2 users, load average: 0.07, 0.00, 0.00
web3: 18:36:56 up 82 days, 22:47, 3 users, load average: 0.04, 0.00, 0.00
web4: 18:36:56 up 82 days, 22:47, 2 users, load average: 0.03, 0.00, 0.00
web5: 18:36:56 up 82 days, 22:45, 2 users, load average: 0.01, 0.00, 0.00
web6: 18:36:56 up 82 days, 22:45, 2 users, load average: 0.01, 0.00, 0.00

# Mac
$ for n in `echo "1 2 3 4 5 6"`; do echo -n "web${n}"; ssh web$n uptime; done;
web1: 18:36:56 up 82 days, 22:50, 4 users, load average: 0.03, 0.00, 0.00
web2: 18:36:56 up 82 days, 22:48, 2 users, load average: 0.07, 0.00, 0.00
web3: 18:36:56 up 82 days, 22:47, 3 users, load average: 0.04, 0.00, 0.00
web4: 18:36:56 up 82 days, 22:47, 2 users, load average: 0.03, 0.00, 0.00
web5: 18:36:56 up 82 days, 22:45, 2 users, load average: 0.01, 0.00, 0.00
web6: 18:36:56 up 82 days, 22:45, 2 users, load average: 0.01, 0.00, 0.00

That is no longer the case. I could have written this myself, but since I am not for reinventing the wheel, I Google’d around and found count.

$ for n in `count 1 6`; do echo -n "web${n}"; ssh web$n uptime; done;
web1: 18:36:56 up 82 days, 22:50, 4 users, load average: 0.03, 0.00, 0.00
web2: 18:36:56 up 82 days, 22:48, 2 users, load average: 0.07, 0.00, 0.00
web3: 18:36:56 up 82 days, 22:47, 3 users, load average: 0.04, 0.00, 0.00
web4: 18:36:56 up 82 days, 22:47, 2 users, load average: 0.03, 0.00, 0.00
web5: 18:36:56 up 82 days, 22:45, 2 users, load average: 0.01, 0.00, 0.00
web6: 18:36:56 up 82 days, 22:45, 2 users, load average: 0.01, 0.00, 0.00

I can’t take any credit for this script, nor do I plan to. But I feel that all Mac shell users should have this available to them. The original was written by Dave Taylor and is located here. I strongly suggest that in addition to copying the script, you read his explanation of its utility. It give you the ability to increment or decrement, whichever you find the need for.

To install, just copy and paste into /usr/local/bin/count and then chmod +x /usr/local/bin/count.

Here is the script in its entirety.

#!/bin/sh

# count - step through numeric values until you get to the max value
if [ "$#" -lt 2 ] ; then
echo "Usage: count {increment}"
exit 1
fi

counter="$1"
max="$2"

if [ "$#" -eq 2 ] ; then
step=1
else
step=$3
fi

while [ $counter -le $max ] ; do
echo $counter
counter=$( expr $counter + $step )
done

exit 0

Git Branch Name in Your Bash Prompt

eric — Fri, 11 Dec 2009 11:50:32 +0000

I work with a few repositories at any given time. And during that time, I typically have multiple branches created for each repository. I figured that it would make my life easier if I knew which branch and/or repository I was working in. Luckily, very little hackery is required here since the git distribution already comes with such a tool. (Note: If you didn’t build Git from source, then you may not have this file.)

First thing I did was copy the git auto-completion file into my home directory:

elubow@beacon repo$ cp /usr/local/src/git/contrib/git-completion.bash ~/.git-completion.sh

Then I edited my .bashrc file to load up my new PS1 variable (PS1 is variable name for the Bash prompt) with git information only if we are currently in a repository. Don’t forget to set a default PS1 in case your .bashrc can’t find your .git-completion.sh file. Another item of note is that you should have the if state below load after your load your PATH variable. This removes the potential for git having to be located after every commend.

# Set the base PS1
PS1="\u:\h \W$ "

# Source the git bash completion file
if [ -f ~/.git-completion.sh ]; then
source ~/.git-completion.sh
GIT_PS1_SHOWDIRTYSTATE=true
PS1='\u@\h$(__git_ps1 " (%s)") \W$ '
fi

export PS1

You’ll notice I also set a variable called GIT_PS1_SHOWDIRTYSTATE. What this variable does is it adds a * to the branch name if the branch has been changed.

There are other options here if you are looking for something more in depth. Git Prompt provides you with a much more in depth version on the command line fo your dirty files and is much more visually oriented.

Additionally, you now get to make use of the other auto-complete features that this script was actually intended for.

elubow@beacon (master*) repo$ git sta
stage stash status

Note the difference between my prompt at the beginning of the post and the end. When I am not in a repository, nothing shows in my PS1 variable.

It’s just that simple to do this and definitely saves some time. Enjoy.

Printing New Lines in Bash

eric — Thu, 22 Oct 2009 15:00:55 +0000

Ran across this the other day and decided it required sharing. If you want to print a new line ‘\n‘ in an echo statement in bash, one would think its just as simple as:

beacon:~ elubow$ echo "This is a test\n"
This is a test\n

The problem is that this doesn’t interpolate the newline character. (For more information on interpolation, see Wikipedia here.) In order to have the newline interpolated, you need to add the command line switch ‘-e‘.

beacon:~ elubow$ echo -e "This is a test\n"
This is a test

This will force Bash to interpolate any non-literal characters in the quotes. Note: Unlike Perl, single or double quotes don’t matter here when Bash is deciding whether or not to interpolate the new line characters.

Quick Date Calculations With Date

eric — Tue, 21 Jul 2009 14:00:29 +0000

I frequently find myself needing to do quick date calculations in order to make scripts run when I want them to or how I want them to. Usually Date::Calc is just a bit too heavily, especially if it’s something as simple as a BASH script. As it happens, date is quite a powerful tool for some command line fu.

For example, to find the first and last day of last month:

FIRST_DOLM=`date -d "-1 month -$(($(date +%d)-1)) days" "+%Y-%m-%d"`
LAST_DOLM=`date -d "-$(date +%d) days" "+%Y-%m-%d"`

You can do the same thing using the date command on a mac with a slightly different set of switches:

FIRST_DOLM=`/bin/date -v1d -v-1m "+%Y-%m-%d"`
LAST_DOLM=`/bin/date -v31d -v-1m "+%Y-%m-%d"`

Both of these will produce the same results:

2009-06-01
2009-06-30

No related posts.

Deleting Lots Of Files (SysAdmin problem solving exercise)

eric — Mon, 17 Dec 2007 11:15:12 +0000

Since I know I am not the first (or the last) to make a typo in logrotate and not catch it for a while…someone else must have been in the position of having to delete a lot of files in the same manner. I recently learned that, as usual, there is more than one way to handle it.

To put the situation in context, I basically allowed thousands of mail.* files to be created. These files littered the /var/log/ directory and basically slowed down the entire file system access. I figured out this a number of ways.

The first way was when I tried to do an ls anywhere, it would just hang. My first reaction was to check to see what was eating up the CPU. To do this, I did a top. I noticed that logrotate was hogging all the CPU cycles. Since I know that logrotate basically only operates on one parent directory (by default) /var/log, I headed on over there and did an ls. Once again, it just hung. Then I figured the file system was slow and decided to check out some file system information. The next two commands I ran were df -h and df -i. I ran the df -h to see if we were out of disk space (and yes I lazily use human readable format). I ran the second to check to see how many inodes were in use. (For more information on inodes, check out the wikipedia entry here).

Now that I know the system is short on inodes, I checked out the output of lsof. Now I know that we have some serious problems in the /var/log dir. After some quick investigation, I realized that there were too many mail.* files. How do I get rid of them? Glad you asked… Let’s assume that we want to delete ALL the mail.* files in the /var/log directory.

1) The easiest way is to do it with find:
1a) Using find‘s delete command:

[root@eric] /var/log # find ./ -type f -name "mail.*" -delete

or
1b) using find‘s exec command with rm:

[root@eric] /var/log # find ./ -type f -name "mail.*" -exec rm -rf '{}' \;

These will work, but either will be slow since they doesn’t do batch execution.

2) A slightly more preferred way is to use bash:

[root@eric] /var/log # for n in mail.*; do rm -v $n; done;

This is a little faster, but will still be relatively slow since there is no batch execution. (Note: The -v in the rm will cause quite a bit of output since it is showing you EVERY file it deletes. Feel free to leave this out if you really screwed up.)

3) The actual preferred method is to use find:

[root@eric] /var/log # find ./ -type f -name "mail.*" | xargs rm -f

I believe this is the preferred method because although it removes the files one at a time, it is more efficient for the file system since it batches it up.

There are certainly other ways to accomplish this task. It can always be done with a Perl one-liner or even using some Perl modules to save some time. These are just a few ideas to point someone in the right direction.