Archive for the ‘ Programming ’ Category

Adding Features to NSQ

Monday, December 23rd, 2013

After being a fairly heavy user of NSQ over the past year or so and finding that it was missing a few features, I decided to jump in and try to add them myself. The only issue was that I didn’t know Go. Since something as simple as not knowing the language the application was written in has never stopped me before, I wasn’t going to let it stop me now.
(more…)

Learning to Hardware Hack at RobotsConf

Monday, December 9th, 2013

I’ve been a programmer (if you can call me that) for quite a few years now. But for the most part, it’s really always been about designing software based systems. Even though these systems are larger than the average startup or SaaS company would get to work with, it’s (as I said) still about designing software based systems. Enter Robots Conf. (more…)

Common Pig One Liners

Tuesday, March 1st, 2011

As with any programming language, there is a bit of a learning curve with Pig. So here are a few common items that I found useful. If you know Pig, please feel free to add your own in the comments section.
(more…)

Pig Queries Parsing JSON on Amazons Elastic Map Reduce Using S3 Data

Wednesday, February 23rd, 2011

I know the title of this post is a mouthful, but it’s the fun of pushing envelope of existing technologies. What I am looking to do is take my log data stored on S3 (which is in compressed JSON format) and run queries against it. In order to not have to learn everything about setting up Hadoop and still have the ability to leverage the power of Hadoop’s distributed data processing framework and not have to learn how to write map reduce jobs and … (this could go on for a while so I’ll just stop here). For all these reasons, I choose to use Amazon’s Elastic Map infrastructure and Pig.
(more…)

Benchmarking in jRuby NYC.RB Talk

Wednesday, November 10th, 2010

Here are the slides from my presentation on jRuby during the NYC.rb talk on 11/9.

NYC.rb – Nov 2010 – Benchmarking jRuby

JSON Benchmarks in jRuby

Monday, October 25th, 2010

I am in the process of switching a major application from MRI Ruby (specifically 1.8.7-p302) using many C extensions to jRuby (currently trying 1.5.3-master). In my application, performance is extremely important. It is so important in fact, that I will be writing about some of my experiences in troubleshooting the speed and getting those important milliseconds back. When I am trying to keep an entire transaction from start to finish under 40ms and just the decoding of a JSON object into a Ruby object in jRuby takes roughly 30ms using json_pure, we may have to explore other avenues.
(more…)

Hash Autovivification in Ruby

Wednesday, October 6th, 2010

One of the features that I miss most from my Perl days (and to be honest, there isn’t a whole lot I miss from my Perl days) is autovivification. For more information on what it is, read the wikipedia page on it here.
(more…)

Interesting Object Methods in Ruby

Monday, September 27th, 2010

This little Rubyism is something that I use frequently for debugging my objects. I add a method to every object to show only the interesting methods. What do I mean by interesting methods?
(more…)

Stopping Curb From Segfaulting

Thursday, July 8th, 2010

While trying to get the curb gem up and running using Ruby 1.8.7p174, I kept getting segmentation faults. I Google’d around and really wasn’t able to come up with much other than lots of people saying not to use ports here. Since I don’t use Mac Ports, I use Homebrew, I figured this wasn’t an issue. I had also recently installed the latest XCode so I incorrectly assumed there was no issue there either. (Note: I am running on a freshly upgraded version of Mac OS X Snow Leopard).
(more…)

Creating Configuration Files With Ruby Templates

Wednesday, June 30th, 2010

I recently had a very repetitive configuration file that needed creating. There were approximately 50 config blocks of 10 lines each with only the host name changing with each block. So I decided to take a shortcut and do it in Ruby using ERB templates. This is so easy and literally save me hours worth of work.
(more…)

Using Unique Keys and Key Groups with Background Jobs in Gearman::Client

Wednesday, June 16th, 2010

While diving into Gearman using Gearman::Client with MySQL and libdrizzle (I know, a mouthful), I ran into what I thought was a bug. I was only able to add 1 background job of any type at a particular time. The launchpad “bug note,” which is available in its entirety here, is rightly labeled won’t fix.
(more…)

Multiple Input Locations From Bash Into Ruby

Thursday, May 13th, 2010

I have been trying to figure out how, while using OptionParser to be able to check for files being input on the command line and if they don’t exist, check other input streams (like Bash). This initially wasn’t very easy since input streams are blocking. So with a little help from friends (thanks roberto), I was able to use his method of non-blocking IO and wrap it in a begin/rescue block. I also took a little advice given in this Stack Overflow question called Best Practices with STDIN in Ruby.
(more…)

Should I Mock Kernel#exit

Monday, April 26th, 2010

I don’t know what the right answer is, so any insight would be appreciated. From Googling around, I came across this thread on StackOverflow which talks about Validating exits and aborts in Rspec. I have a few issues with this, namely that I am using shoulda and I don’t want to validate the exit, I want to stop it from happening inside of the test.
(more…)

ColorTail Gem

Tuesday, April 20th, 2010

One of the best ways to learn something new is to find a project in that realm and do it. I wanted to learn Ruby on Rails, so I wrote a web site. I wanted to learn Python, so I did a project in it for work. Now I want to get better at Ruby, so I wrote a gem. It’s usually not as simple as just picking a project though. It’s much easier to pick and complete a project if you are filling a need. So that’s what I did, I filled a need.
(more…)

Monitoring Services with Nagios::Plugin

Wednesday, April 7th, 2010

There are a lot of people who say, “if it isn’t monitored, then it isn’t a service.” The problem is that I don’t think enough people outside of the systems world believe that or even understand why its said. I think the primary offenders here are developers. It isn’t because they don’t know better, but typically developers just want to get the application up and running and then move on to developing the next thing. I also think there is some fault on the side of the administrators and the managers not insisting that part of the completed version of a project includes monitoring. But I don’t want to harp on this as much as I would like to show just how easy it is to compensate here by taking advantage of Nagios::Plugin.
(more…)

Database Read/Write Splitting in Frameworks/ORMs

Monday, March 1st, 2010

Although one of the primary ideas behind frameworks is to keep things as simple as possible, sometimes they create issues in the long run. What I am about to discuss is something of a luxury problem (as scaling usually is), but it is a problem nonetheless.

When initially starting a project, whether you are using Ruby on Rails (Ruby), Django (Python), CakePHP (PHP), Catalyst (Perl), or any of the other 100s of frameworks in any of the languages out there, the first and most important thing to do is to get it out the door. Once you have done that, it’s time to get users, fix bugs, and add features. After you have done all that and you have a great web app, its time to think scaling. (Yes I realize that I have trivialized this process immensely, but its for a point, I promise).
(more…)

Textmate Minimap Plugin

Thursday, February 25th, 2010

My editor of choice for most of what I do on the Mac (when I’m not in the shell using Vim) is Textmate. I recently came across a neat plugin called the Textmate Minimap. Essentially this plugin opens a map across the side of your Textmate window (like another drawer on the other side) and allows you to scroll full screens. You can move the editing window screen by screen in the minimap. This is equivalent to a visual page down.
(more…)

Philosophical Python

Friday, February 19th, 2010

Having spent a lot of time in the Perl world and even trying to write a Perl poem here and there, I decided to give some Python poetry a try. It turned out to be a bit more philosophical than poetic. But it compiles and that’s what counts; even though it doesn’t do anything useful (or anything at all).
(more…)

Model Specific Formatted Search Results Using Thinking Sphinx

Monday, February 8th, 2010

Having recently implemented Thinking Sphinx on one of my web sites, I thought it would be cool to be able to search every indexed model. With Thinking Sphinx, it’s easy to have a bunch of different classes returned in the results. The tougher part is displaying them in a way that is organized (although admittedly not very DRY).
(more…)

When To Use MySQL Cursor Classes In Python

Monday, January 18th, 2010

I have been writing a lot of code that has been interacting with MySQL lately. Sometimes I find it easier to work the result set in a dictionary form and other times it is easier with an array. But in order to not break all your code, it is necessary to set a default cursor class that keeps your code consistent. More often than not, I find using using a arrays is easier since I just want quick access to all the retrieved data. I also end up making my SELECT calls while specifying the columns and order of the columns I want returned.

The reason that using cursor classes is handy is because Python doesn’t come with a mysql_fetch_assoc like PHP or selectrow_hashref like Perl’s DBI interface. Python uses cursor dictionaries to bridge this gap. Ultimately your result is the same. But as with Perl and PHP, defaulting to cursor dictionaries isn’t a good idea for larger datasets because of the extra processing time and memory required to convert the data.
(more…)

Python Multiprocessing Pools and MySQL

Monday, December 21st, 2009

There really isn’t a solid Python module for multiprocessing and MySQL. Now this may be because MySQL on a single server is disk bound and therefore limited in speed or just because no one has written it. So here is a quick and dirty example using the Pool module in multiprocessing in Python 2.6 and MySQLdb.

I also tried using PySQLPool. This was designed for threading and not forking as I am doing with Pool method. Although I am sure it is possible to use PySQLPool with forking by passing the connection (pool) object down to the child process or possibly doing something with IPC, I decided to keep it simple (although slightly more expensive) and instantiate MySQLdb connections upon fork.
(more…)

Python's MySQLdb 2014 Error – Commands out of sync

Friday, December 18th, 2009

While writing a simple Python script to access and process data in a database, I came across an error that said:

1
Error 2014: Commands out of sync; you can't run this command now

After quite a bit of Googling and with very little findings, I had to dive in a little and try to figure out what was going on. The whole error looked like this:
(more…)

Custom Google Maps Marker With YM4R_GM

Monday, December 14th, 2009

In one my Rails applications, I allow the user to search for surrounding businesses from their current location. I always showed them a You Are Here marker. The issue I had with this was that the marker was always the icon as the search results. Differentiating these markers is actually extremely easy with ym4r_gm plugin.

First thing is to find a custom icon that you want to use. You can just Google for custom Google maps icons. I chose to use their default icon, just in a different blue. (You can download it here so you are working with what I am working with for this example). The next thing I did was to use the Google custom markers web site to find the proper config options for the icon.
(more…)

Country-State Select Using Carmen and jQuery

Monday, December 7th, 2009

I’ve been wanting to find a way to use drop down menus for countries and their states when they exist. But keeping a list on my own would have been a huge pain in the ass. So rather reinvent the wheel, I found the Carmen plugin for Rails. All I have to do is keep the plugin updated and my list of countries and their states will be kept updated as well.

How do I do all this with unobtrusive Javascript and Rails you ask? Good question. Let me show you. Don’t forget to install the plugin (or use the gem).

Let’s start out by adding the drop down menu to our view. In my case I have it in a partial for the address part of a form. You’ll have to modify this slightly to pick up the values of the form if the partial handles edits as well. This one is just for a new method as it uses a default country of US and its states. Note the div ID here of addressStates; we will be using this later in the javascript.
(more…)

Adding AJAX Bookmarks to Your Rails Application (Part 2 of 2)

Wednesday, November 25th, 2009

In part 1 of this series, we discussed the base models, controller, database migrations necessary to get this project off the ground. Now we are going to continue with this functionality

Let’s take a look at what needs to go into the models to support this. If you have a model that uses a slug generated via to_param, then your code will look like the top model, If you are using the normal numeric id convention, then it will look like the bottom model. The reason for the specifically named methods get_title and get_description will become apparent when you start displaying bookmarks. The thought process is that you can use a consistent set of calls for displaying the bookmark information and put the code to grab that information in the model where it belongs rather than loading up the helper methods. What should also be noted is that the title and description fields are not always consistent across models. Therefore the method naming conventions returns the proper column with consistent method names.
(more…)

Adding AJAX Bookmarks to Your Rails Application (Part 1 of 2)

Monday, November 23rd, 2009

It you want to add the ability to bookmark pages in your Rails application, its actually a fairly straightforward thing to do. You can even do them in AJAX. There may be better ways to do this, but this way is somewhat abstract and it works for me, so hopefully it can work for you too. It is abstract in the sense that it will work for models with different URL styles and different column names.

The way this works is that you add a bookmark icon (which is initially disabled) to a show <model_name> page. When the user clicks on the bookmark icon, an AJAX query will be made in the background and update the users bookmark lists. I am approaching this from an abstract methodology. Meaning that I have “forced” these methods to work with models executed in various fashions (as I give examples of below). The AJAX call is going to be simply work as a toggle. It will actually call a toggle method in the bookmarks controller and change the current value and replace the image. The user can then view the pages they have bookmarked in their profile.

I have decided to break this into a multi-part blog entry because it ends up being quite long. Not necessarily in how long it takes, just the amount of space it takes to show all the code. I have done my best to only show relevant code and maintain brevity. Note: I will not cover how to allow for unobtrusive AJAX calls. That is beyond the scope of this set of posts.
(more…)

File Read Write Create with IO::File

Friday, November 20th, 2009

Ran into an annoying gotchya with Perl’s IO::File. Apparently opening the file in append mode with read access if the file already exists puts the file position pointer at the end of the file. If it doesn’t exist, it creates the file. Note the +>>, that opens the file r/w/append. You can also use the more common (and more easily recognizable) form of a+.

1
2
3
4
5
    my $FH = new IO::File "$file", "+>>";
    while (my $line = $FH->getline()) {
      print "Line: $line\n";
    }
    undef $FH;

I noticed that when I tried to read the file (if it already existed), then nothing would be read. I neglected to realize that you must seek to position 0 in the file if you want to read it. Therefore the following code will work:

1
2
3
4
5
6
    my $FH = new IO::File "$file", "+>>";
    $FH->seek(0,0);
    while (my $line = $FH->getline()) {
      print "Line: $line\n";
    }
    undef $FH;

Although it might seem obvious that you need to be at the beginning of the file to read it forward (and it is), I didn’t realize the file pointer opened a file in append mode to the last position in the file (in hind sight, it does appear to be a bit more obvious).

One Time Modal Windows With Rails and Fancybox

Tuesday, November 3rd, 2009

Let’s say that you have a situation that you want to have a modal window show up only once for each user. It’s actually not that difficult although lots of Googling around got me nowhere. I am choosing to use FancyBox for my modal window, but feel free to use your modal framework of choice. So let’s get down to business.

First thing you’ll need to do is download FancyBox and copy the stylesheets, images, and Javascript files to their proper/desired location in your Rails app. Style the window according to your likings.

Whether it is right or wrong, I did this entirely in the view, without even pulling the Javascript out into the application.js (or even another Javascript file for that matter). My reason was that I only want the modal window showing up on this page. If you want your modal window to show up somewhere else (or on every page), then put the code in your layout. But remember that this call will be executed every time the page loads. I put mine in a profile page which doesn’t get accessed that often so the conditional is not checked quite as frequently.

My application uses Facebook Connect and grabs the users Facebook Proxy email address (FB app developers will know what this is). So I check if that’s the email I have for the user. If yes, then I pop up a modal window on page load only once to get their regular email address and possibly a password so they can login to their account without Facebook Connect if they want. When the modal window is shown, a variable is set in the cookie (note that this cookie is shared with authlogic for sessions) to ensure that the modal window isn’t shown again.
(more…)

Printing New Lines in Bash

Thursday, October 22nd, 2009

Ran across this the other day and decided it required sharing. If you want to print a new line ‘\n‘ in an echo statement in bash, one would think its just as simple as:

1
2
beacon:~ elubow$ echo "This is a test\n"
This is a test\n

The problem is that this doesn’t interpolate the newline character. (For more information on interpolation, see Wikipedia here.) In order to have the newline interpolated, you need to add the command line switch ‘-e‘.

1
2
beacon:~ elubow$ echo -e "This is a test\n"
This is a test

This will force Bash to interpolate any non-literal characters in the quotes. Note: Unlike Perl, single or double quotes don’t matter here when Bash is deciding whether or not to interpolate the new line characters.

Migrations Without belongs_to Or references

Wednesday, October 14th, 2009

Normally when do a database migration in Rails, when adding ownership from a model to another model, you use the concept of belongs_to or references:

1
2
3
4
  create_table :comments do |t|
    t.belongs_to :user
    t.references :post
  end

Interestingly enough, these methods are only available during the initial table creation. If you want to add a reference to a model that is created later, you have to do it the old fashioned way, by just adding a column:

1
   add_column :comments, :group_id, :integer

Doing it this way is clean, easy, and definitely meets the KISS principle. But I do find it interesting that one can’t add an association later in the game. Sometimes the Rails way is just KISS and adding the column by hand.