Lots of people forget about the usefulness of the core utilities (the tools available in Bash). I am even pretty guilty of it at times with such quick and easy things like Perl, Ruby, or Python that allow you to process items from the command line. However, they load up an entire interpreter. It is usually better to use the coreutils.

I’ll give you the specific example I had to deal with, but this can be extrapolated out and I’m sure reused for other purposes. I have lists of email addresses of who has received emails that were sent over the past 7 days. I do this with a find command since the address files are created nightly. They are laid out with 1 email address (and some other meta data) per line. You’ll notice the awk in the find command; that’s just to extract the email address from the line to make things easier to work with.

The overall goal here is find out how many times people were emailed over the past 7 days. Another way of saying it is, how many people received 1,2,3,4, etc mailings over the past week. This is an exercise in aggregation.

First, here is everything in its entirety and then I will proceed to go through it all piece by piece.

A quick breakdown goes as follows.

1) Find all mailing list master files (-name) that aren’t archived (-maxdepth) that were created in the last 7 days (-ctime) and print their id (exec awk) to a tmp file. Everything here needs to be concatenated so as not to clobber the emails added by the previous day.

2) The next piece word frequency counts the emails and prints them to STDOUT in a manner. This is to say that the first column is a count of how many times the second column has received an email over the past 7 days. Now we know how many emails each individual has received and we need to aggregate again.

3) Because I am using pipes, instead of printing the previous step to STDOUT, I only print the first column (count) and then count the frequency of that.

4) Then I swap the columns so I get . This is much more logical and is the way we generally read tables.

5) Finally I sort that result numerically ascending and remove the temp file so I don't cat things to it again.

6) We get our output in an easy to read format (Note: I added the headers).

Related posts:

1. Counting Email Addresses By Domain in MySQL
2. Email::Find
3. Count Instead of Sequence

One of my issues with MongoDB is that, as of this writing, there is no way to retrieve a random record. In SQL, you can simply do something similar to “ORDER BY RAND()” (this varies depending on your flavor) and you can retrieve random records (at a slightly expensive query cost). There is not yet an equivalent in MongoDB because of its sequential access nature. There is a purely Javascript method in the MongoDB cookbook here. If you are really interested, I would also read the Jira ticket thread #533 on this issue.

Although it feels a little dirty and kind of hackish, here is how I accomplished getting a random record using the Mongo-Ruby driver. Part of this is documented in the cookbook article I linked to above, but I reiterate bits and pieces of it here. This is essentially the same thing that any “ORDER BY RAND()” statement is doing, its just not doing it “on the fly”.

The first thing you’ll have to do is add an additional column to the collection; we’ll call it random. For the ease of use, we’ll also say that every value that goes in this column is between 0 and 1 (and can therefore be generated via Kernel.rand()). This is important because we are going to use it as our criteria for finding a random record.

First, initialized the connection to the database and bind an instance variable to a collection. Then generate the random number that you are going to use to find a random record. Now we try to find_one document that is greater than or equal to our random number. In case we miss, we also do a less than or equal to next. This means that as long as we have at least 1 document in our collection, we will return a record. The more documents in the collection, the better the randomness of the returned document.

For reference, a mongodb collection with a random column may look like this:

No related posts.

Anyone who has spent any time in a shell and has been cut off while working should know about screen. If not, then I recommend reading up on it (here or here). But I’m not here to tell you about screen as a general tool, I want to show you how to use it for screen sharing. I found a couple of forum posts and other scattered information, so here’s a little centralizing of information.

First a definition, sharing a screen session means both users can at the very least view the same screen (console) at the same time. And depending on the security settings, they can both type as well (which can be very annoying depending on who your sharing the screen session with). I’m going to assume you know the basics so we’ll go on from there.

Let’s assume that I elubow want to share my console with jdoe. After screen is installed, these next 2 steps are critical otherwise the sessions will not be accessible to other users regardless of which commands you issue inside of the screen session:

The easiest way to do this next part is to actually name your screen session. So let’s call our session test.

This will create the session and put you in it. Now we need to add the access control. Note: You can automate the multiuser piece by adding it to you .screenrc.

All jdoe has to do now is connect to the same machine and type the following command:

There are other security combinations as the above gives jdoe full access. Here are a few common ones:

Related posts:

1. Cluster SSH with cSSHx
2. 10 Tips To Start Securing Your Linux System
3. SSH Organization Tips

While trying to get the curb gem up and running using Ruby 1.8.7p174, I kept getting segmentation faults. I Google’d around and really wasn’t able to come up with much other than lots of people saying not to use ports here. Since I don’t use Mac Ports, I use Homebrew, I figured this wasn’t an issue. I had also recently installed the latest XCode so I incorrectly assumed there was no issue there either. (Note: I am running on a freshly upgraded version of Mac OS X Snow Leopard).

Since everything built correctly, I just assumed it would work correctly. Here it is obviously not doing so.

So I narrowed it down to what I believe is an issue with the curl headers. The solution is to install a new curl library. Since I wanted to do it the Homebrew way, and Homebrew doesn’t have curl by default, here is what I did.

Copy the Homebrew Formula code from http://github.com/adamv/homebrew/blob/duplicates/Library/Formula/curl.rb into a file named curl.rb in your formula directory (mine is /usr/local/Library/Formula/). You should then be able to:

Don’t forget to logout and log back into the shell so the new version of curl is available:

Now you have to uninstall and reinstall the curb gem. The easiest way that I found (since I have bundler installed) was to do bundle show curb to find out where the gem is located. I removed the directory and reran bundle install (or gem install curb if you don’t have bundler).

Because I am using rvm (Ruby Version Manager), I actually encountered this issue with 1.9.1p378 as well. This fixed the issue for both versions.

No related posts.

I recently had a very repetitive configuration file that needed creating. There were approximately 50 config blocks of 10 lines each with only the host name changing with each block. So I decided to take a shortcut and do it in Ruby using ERB templates. This is so easy and literally save me hours worth of work.

I started out by creating a template for the rsyslog block I wanted to replicate. In the full version of the script, there is an additional section for the access.log file which you’ll see below. If you know Rails, then this template style should look very familiar.

The below block of code creates a configuration block (above) for each of virtual hosts in the array and then prints it to the screen.

This is going to produce the following block which you can then pipe to a file:

Here is the script in its entirety:

Related posts:

1. Parsing Ini Files With Ruby
2. Multiple Input Locations From Bash Into Ruby
3. Creating Dummy Packages On Debian

Since Google Buzz is Twitterish in the sense that you can post a quick note, but different in the sense that (amongst other things), it can be longer than 140 characters. So in that vein, I’m starting to try to make a habit of a quick post (a couple per week) of things I do to make my life easier. This goes both for SysAdmins and for Programmers.

I’ll be adding things for everything from Vim and Bash, to MySQL, Perl, Ruby, Python, and any other language or app that I happen to be using shortcuts in. If you have another idea or a better way to do something, I’d love to hear it. So add a comment or hit me up on Twitter (@elubow). I’ve already got a few up there so check them out.

And for the record, I am not trying to mimic or one up something like commandlinefu.com. Just providing things that work well in my world.

The Google Buzz account is here.

No related posts.

While diving into Gearman using Gearman::Client with MySQL and libdrizzle (I know, a mouthful), I ran into what I thought was a bug. I was only able to add 1 background job of any type at a particular time. The launchpad “bug note,” which is available in its entirety here, is rightly labeled won’t fix.

Tracking that post down wasn’t too difficult, but it would have saved me time if that information was in the Perl documentation. Using Gearman::Client you can create your unique ID (or UUID). To create a background job, your code would then look like this:

This will ensure that you have a unique UUID when INSERTing the job. But I think that we can do better and this is where the good practice comes into effect (and why the Gearman devs marked it as won’t fix). So let’s think a little more abstractly for a moment. For the uniq key (unique_key column), how about something like “${type}_”.create_uuid_as_string(). This will allow us to group (in this case) our mailing types so that we can monitor the status of our work queue. Yes, we can easily hop into the database and run this query:

But wouldn’t it be nicer to know exactly where we are at with this query:

For reference, here are the unique_key columns that allowed to generate that query:

Related posts:

1. MySQL Encoded URI Search and Replace
2. HOWTO Recreate /dev/null
3. Speeding Up Your Selects and Sorts

The Drobo is supposed to be one of those solutions that essentially pops out of the box and with very little effort, just works. I ordered my Drobo FS through an Amazon retailer. What I wasn’t expecting was an experience…

The whole “Drobo Experience” starts out when you open up the box it’s shipped in. The first thing you see is a box that reads, “Welcome to the World of…” What a great way to start. This would make Seth Godin proud. The actual unit was wrapped in a black cloth that said Drobo. In fact, when you remove the top box that says, “welcome to the world of”, you see a black box that says Drobo. That’s just cool. The front even has a magnetic cover (very weak magnets in order to not affect the drives) in the front to provide a clean look.

Now on to the actual product. This could not have been easier to get going. I had myself all geared for a few hours of work. Instead, I just threw the drives in, powered them on and was in business. And the fact that the Drobo now natively supports Time Machine made it all that much easier. One thing I did do is download a Drobo app that limits the size of a Time Machine Partition. This ensures that Time machine doesn’t eat up your entire Drobo partition (which Time Machine can do quickly if left unattended).

Now that the important stuff is out of the way and I have the Drobo setup with Time Machine, I moved onto setting up DroboApps. Again, this could not have been any simpler. Just mount the DroboApps share and copy the .tgz files to it and restart the Drobo. Boom! Done deal.

This has seriously been one of the most pleasurable hardware experiences I have had in a long time. I wish someone at Data Robotics was paying or something for this glowing recommendation, but no such luck. This is just one of those products that gets its right. Remember the cliche, “you get what you pay for?” Well this is a perfect example.

No related posts.

I have been trying to figure out how, while using OptionParser to be able to check for files being input on the command line and if they don’t exist, check other input streams (like Bash). This initially wasn’t very easy since input streams are blocking. So with a little help from friends (thanks roberto), I was able to use his method of non-blocking IO and wrap it in a begin/rescue block. I also took a little advice given in this Stack Overflow question called Best Practices with STDIN in Ruby.

First we need to get the file list off the command line and assume that anything left are files.

Since we are using threads, I open up a thread for STDIN and killed it when we run out of input.

Now its time for the files.

Using these code snippets, you will be able to use input from both files on the command line and STDIN:

1. $ myscript.rb file1 file2
2. $ cat foo | myscript.rb file1 file2
3. $ myscript.rb file1 file2 < foo

Related posts:

1. Should I Mock Kernel#exit
2. Parsing Ini Files With Ruby
3. Git Branch Name in Your Bash Prompt

Since one of the core communication methods for my company amongst engineers is 37Signals Campfire and Nagios is one of our main monitoring tools for all of our applications and services, I thought it would be a good idea to combine the two. So with a few simple additions to the Nagios configuration and a Ruby Campfire script, you can get this up and running.

On the Campfire side, I will leave the exercise of adding a Campfire user up to you. You may need to go through your Campfire administrator for that.

On the Nagios side, the first thing that needs to happen is that you need to add the following commands to your command.cfg file in your Nagios configuration. Make sure that you specify the proper location of your notify-by-campfire.rb file.

Add this contact template definition for the Campfire contact to your contacts.cfg. Don’t forget to add it to a contactgroup.

The Ruby code for notify-by-campfire.rb is here. Just install it wherever you’d like and as I said above, make sure its specified properly in your commands.cfg.

Note: I am aware that I could have made this script much shorter, but my goal is to make this extensible should I want to provide more information or change the format. Feel free to modify for your own usage.

Update (2010-05-17): To make it a little easier to read in the context of a smaller Campfire window, I have amended the notify-host-by-campfire and notify-service-by-campfire entries to be shorter:

Related posts:

1. Monitoring Services with Nagios::Plugin
2. Sys::Hostname