Erics Tech Blog » database

ec2-consistent-snapshot With Mongo

eric — Thu, 21 Apr 2011 07:00:47 +0000

I setup MongoDB on my Amazon EC2 instance knowing full well that it would have to be backed up at some point. I also knew that by using XFS, I could take advantage of filesystem freezing in a similar fashion to LVM snapshots. I had remembered reading about backups on XFS with MySQL being done with ec2-consistent-snapshot. As with any piece of open source software, it just took a little tweaking to make it do what I wanted it to do.

Out of the box, ec2-consistent-snapshot works great for freezing an XFS filesystem with MySQL because it not only stops the server, but handles potential replication issues. By following the steps outlined here by 10gen, I just made a few slight adjustments to the core ec2-consistent snapshot script to allow for MongoDB support. In fact, it supports locking and fsyncing immediately prior to freezing and backup. I have been using this script in production for a while now and it seems to work without issue for me.

In the usual spirit of social coding, I have added the script to Github: https://github.com/elubow/ec2-consistent-snapshot.

Running it is just this:

ec2-consistent-snapshot \
--mongo \
--xfs-filesystem /data \
--region us-east-1 \
--description "RAID snapshot $(date +'%Y-%m-%d %H:%M:%S')" \
vol-VOL1 vol-VOL2 vol-VOL3 vol-VOL4 vol-VOL5 vol-VOL6 vol-VOL7 vol-VOL8

The options used here (for reference) are telling ec2-consistent-snapshot to use –mongo, on the –xfs-filesystem /data, in the us-east-1 –region (note that it’s just the region and not the availability zone within that region), to be backed up with the listed –description of the specified volumes. You can even throw a –mongo-stop in there to have Mongo stopped before the file system freeze and then restarted after the volumes have been backed up. Don’t forget that you need to set your Amazon keys in you environment variables (AMAZON_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY for your key and secret respectively).

I attempted to keep the usage style consistent with Eric Hammond’s original version, just add Mongo support for it.

Note: I also mentioned this on the mailing list. But given the amount of messages that fly around on the list daily, some folks may have missed it.

References:

ec2-consistent-snapshot blog entry by Eric Hammond
ec2-consistent-snapshot on Github with Mongo DB support
Backing up MongoDB on EC2 (10gen)

Getting a Random Record From a MongoDB Collection

New Massachusetts Security Law Passed For Databases

eric — Tue, 27 Apr 2010 10:00:28 +0000

In case you haven’t heard about the new Massachusetts state law regarding consumer or client information in databases, you can read about it here, at Information Week, or just Google for “Massachusetts data security law”. And if you haven’t read about, then I strongly suggest you do. This is one of those instances where I believe their heart is in the right place, even if the execution/implementation wasn’t perfect.

I get the feeling that Mass will make an example of a few offenders and then hope that the law either gets picked up by other states or federalized. The problem is that this law will only really affect companies that are headquartered in Mass. I am by no means a lawyer, but I don’t believe that its legal for the state of Mass to go after companies that are headquartered in other states.

Now at the very least, this should be a reminder to developers and sysadmins to make sure that data is properly protected, both in storage and transfer, and that is properly managed within an application. Although this is a scary prospect that the government is getting involved in software design and data management, it certainly isn’t the first time (think HIPAA, FIPS testing, and Sarbanes–Oxley). Although those have been helpful, the goals seemed a little clearer.

This law is going to put a pretty heavy imposition on smaller organizations with regard to user education and basic requirements fulfillment for data storage. It’s going to be quite a bit harder to bring a product to market in Mass. Although it seems like the government is doing a service, here they may be doing a disservice to their state economy.

I guess we’ll have to see how this plays out. Thoughts?

Speeding Up Your Selects and Sorts

eric — Thu, 04 Mar 2010 13:30:07 +0000

When you are using a framework, they typically set your VARCHAR size automatically to 255. This is normally fine since you are letting the framework abstract you away from most of the SQL. But if you interact with your SQL, there is a way to get a decent speed increase on your SELECTs and ORDER BYs when you are working with VARCHARs.

The VARCHAR data type is only variable character size for storage, not for sorting and buffering. In fact, since the MySQL optimizer doesn’t know how big the data in that column can be, it has to allocate the maximum size possible for that column. So for sorting and buffering of the name and email columns below would take up 310 bytes per row.

To fix that you should alter the size of your columns. Imagine having the optimizer have to go through an additional 310 bytes for every row. If there are 500k rows in the table time 310 bytes for each row, that can add up in the amount of memory that the optimizer has to use to perform the sorting/buffering.

Consider the following table of businesses:

You can change the name to column sizes to 75 (name), 50 (email), and 100 (url). But realistically you don’t want to chance things getting cut off, so it may be better off to settle on each column here being a VARCHAR(100). Even that would save drastically on the space required to perform a sort and buffer the results.

Model Specific Formatted Search Results Using Thinking Sphinx

eric — Mon, 08 Feb 2010 14:15:23 +0000

Having recently implemented Thinking Sphinx on one of my web sites, I thought it would be cool to be able to search every indexed model. With Thinking Sphinx, it’s easy to have a bunch of different classes returned in the results. The tougher part is displaying them in a way that is organized (although admittedly not very DRY).

Let’s start out with the controller method for search results (app/controllers/search_controller.rb):

def result
if !params[:model].blank?
@model = params[:model]
else
@model = "ThinkingSphinx"
end
@query = params[:search][:query]

@results = Search.model_search(@model, @query)
end

Also, let’s make sure we define the actual search model. I define it in a separate search library (lib/search.rb). This is the relevant snippet of code:

def self.model_search(model, keywords, var = {})
@search_options = { :page => var[:page] || 1,
:per_page => 15 }

@search_options.merge!( :order => "@relevance DESC",
:sort_mode => :expr,
:sort_by => "@weight * @weight")

model.constantize.search(keywords, @search_options)
end

Assuming that we are working with the all models search, where the above model is ThinkingSphinx, let’s iterate over the search results with this code in the view (app/views/search/results.html.erb):

< % if @results.total_entries > 0 %>
<div id="localLocationList">
<ul class="list">
< % @results.each do |result| %>
<li>
< %= display_search_result(result) %>
li>
< % end %>
ul>
<div class="clear">div>
div>
< % end %>

And the view helper (app/helpers/search_helper.rb):

def display_search_result(result)
eval "render :partial => '#{result.class.to_s.downcase.pluralize}/result',
:locals => { :result => result }"
end

The interesting thing about this bit of code is the eval. The eval on each iteration decides which search result partial to display based on the class and then passes the result to the partial for display. So if the result has a class of Business, the partial app/views/businesses/_result.html.erb will be rendered. This is a quick example of a search result partial:

This is useful because all models don’t have the same characteristics. By creating a search result partial for each model type, this can be reused for consistent looking search results around your webapp. If the search is model specific, the same result partial will be used in every iteration over the results. If the search is model agnostic, then you can display your search results in a consistent manner.

When To Use MySQL Cursor Classes In Python

eric — Mon, 18 Jan 2010 15:00:40 +0000

I have been writing a lot of code that has been interacting with MySQL lately. Sometimes I find it easier to work the result set in a dictionary form and other times it is easier with an array. But in order to not break all your code, it is necessary to set a default cursor class that keeps your code consistent. More often than not, I find using using a arrays is easier since I just want quick access to all the retrieved data. I also end up making my SELECT calls while specifying the columns and order of the columns I want returned.

The reason that using cursor classes is handy is because Python doesn’t come with a mysql_fetch_assoc like PHP or selectrow_hashref like Perl’s DBI interface. Python uses cursor dictionaries to bridge this gap. Ultimately your result is the same. But as with Perl and PHP, defaulting to cursor dictionaries isn’t a good idea for larger datasets because of the extra processing time and memory required to convert the data.

import MySQLdb

conn = MySQLdb.Connect(
host='localhost', user='user',
passwd='secret', db='test')
cursor = conn.cursor()
cursor.execute("SELECT this,that FROM foobar")
rows = cursor.fetchall()
cursor.close()
conn.close()

This then allows me to do just call the DictCursor (dictionary based cursor) on a single cursor object for a particular query and leave all the rest of the queries as array based results.

import MySQLdb
import MySQLdb.cursors

conn = MySQLdb.Connect(
host='localhost', user='user',
passwd='secret', db='test')
cursor = conn.cursor(cursorclass=MySQLdb.cursors.DictCursor)
cursor.execute("SELECT this,that FROM foobar")
rows = cursor.fetchall()
for row in rows:
print row['this'], row['that']
cursor.close()
conn.close()

Note: Doing something like changing the connection string to use a default cursor class of dictionary cursor in an existing code base is not a good idea. It will make your results come back as dictionaries when your current code expects arrays. Yes, I did this and had to track it down the hard way. Hopefully this saves you some trouble.

conn = MySQLdb.Connect(
host='localhost', user='user',
passwd='secret', db='test',
cursorclass=MySQLdb.cursors.DictCursor)

Migrations Without belongs_to Or references

eric — Wed, 14 Oct 2009 14:00:00 +0000

Normally when do a database migration in Rails, when adding ownership from a model to another model, you use the concept of belongs_to or references:

create_table :comments do |t|
t.belongs_to :user
t.references :post
end

Interestingly enough, these methods are only available during the initial table creation. If you want to add a reference to a model that is created later, you have to do it the old fashioned way, by just adding a column:

add_column :comments, :group_id, :integer

Doing it this way is clean, easy, and definitely meets the KISS principle. But I do find it interesting that one can’t add an association later in the game. Sometimes the Rails way is just KISS and adding the column by hand.