In the usual spirit of social coding, I have added the script to Github: https://github.com/elubow/ec2-consistent-snapshot.

Running it is just this:

The options used here (for reference) are telling ec2-consistent-snapshot to use –mongo, on the –xfs-filesystem /data, in the us-east-1 –region (note that it’s just the region and not the availability zone within that region), to be backed up with the listed –description of the specified volumes. You can even throw a –mongo-stop in there to have Mongo stopped before the file system freeze and then restarted after the volumes have been backed up. Don’t forget that you need to set your Amazon keys in you environment variables (AMAZON_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY for your key and secret respectively).

I attempted to keep the usage style consistent with Eric Hammond’s original version, just add Mongo support for it.

Note: I also mentioned this on the mailing list. But given the amount of messages that fly around on the list daily, some folks may have missed it.

References:

• ec2-consistent-snapshot blog entry by Eric Hammond
• ec2-consistent-snapshot on Github with Mongo DB support
• Backing up MongoDB on EC2 (10gen)

Related posts:

1. Getting a Random Record From a MongoDB Collection

The first thing you’ll have to do is add an additional column to the collection; we’ll call it random. For the ease of use, we’ll also say that every value that goes in this column is between 0 and 1 (and can therefore be generated via Kernel.rand()). This is important because we are going to use it as our criteria for finding a random record.

First, initialized the connection to the database and bind an instance variable to a collection. Then generate the random number that you are going to use to find a random record. Now we try to find_one document that is greater than or equal to our random number. In case we miss, we also do a less than or equal to next. This means that as long as we have at least 1 document in our collection, we will return a record. The more documents in the collection, the better the randomness of the returned document.

For reference, a mongodb collection with a random column may look like this:

Related posts:

1. ec2-consistent-snapshot With Mongo
2. Random Tech Notes And Buzz Updates

Now at the very least, this should be a reminder to developers and sysadmins to make sure that data is properly protected, both in storage and transfer, and that is properly managed within an application. Although this is a scary prospect that the government is getting involved in software design and data management, it certainly isn’t the first time (think HIPAA, FIPS testing, and Sarbanes–Oxley). Although those have been helpful, the goals seemed a little clearer.

This law is going to put a pretty heavy imposition on smaller organizations with regard to user education and basic requirements fulfillment for data storage. It’s going to be quite a bit harder to bring a product to market in Mass. Although it seems like the government is doing a service, here they may be doing a disservice to their state economy.

I guess we’ll have to see how this plays out. Thoughts?

Related posts:

1. Country-State Select Using Carmen and jQuery
2. Mod-Security 2.5 by Magnus Mischel

The VARCHAR data type is only variable character size for storage, not for sorting and buffering. In fact, since the MySQL optimizer doesn’t know how big the data in that column can be, it has to allocate the maximum size possible for that column. So for sorting and buffering of the name and email columns below would take up 310 bytes per row.

To fix that you should alter the size of your columns. Imagine having the optimizer have to go through an additional 310 bytes for every row. If there are 500k rows in the table time 310 bytes for each row, that can add up in the amount of memory that the optimizer has to use to perform the sorting/buffering.

Consider the following table of businesses:

You can change the name to column sizes to 75 (name), 50 (email), and 100 (url). But realistically you don’t want to chance things getting cut off, so it may be better off to settle on each column here being a VARCHAR(100). Even that would save drastically on the space required to perform a sort and buffer the results.

Related posts:

1. MySQL Encoded URI Search and Replace
2. Using Unique Keys and Key Groups with Background Jobs in Gearman::Client
3. HOWTO Recreate /dev/null

When initially starting a project, whether you are using Ruby on Rails (Ruby), Django (Python), CakePHP (PHP), Catalyst (Perl), or any of the other 100s of frameworks in any of the languages out there, the first and most important thing to do is to get it out the door. Once you have done that, it’s time to get users, fix bugs, and add features. After you have done all that and you have a great web app, its time to think scaling. (Yes I realize that I have trivialized this process immensely, but its for a point, I promise).

When starting to scale (whether its out or up) and you decide its time to add another database, its necessary to analyze your app and decide whether its read heavy or write heavy. A lot of scaling comes in knowing your application and where its bottlenecks are. Let’s assume that you are at the point that you need to add a database server. What would be great is if you had a framework that allowed you to set some database servers as read-only in order to take load off the master.

In an abstract format, it would be a good idea to break out your SQL requirements into 2 functions: sql_write_query and sql_read_query. Then have the functions go to your primary database server and slave database servers respectively. The reason that you should do this instead of using a single function that sends the query to the “correct” location based on the SQL it finds is that your slave servers may be behind the master (which is the nature of replication) which could give you an incorrect result in your query. This way, depending on the importance and type of the query, in your code you can choose the location that you want to send the query to. The read queries where accuracy is extremely important can be sent to the database using sql_write_query and all others can be executed normally using your sql_read_query function.

How does this relate to frameworks and ORMs? It would be very handy if frameworks provided a method to expand an application into splitting read and write queries that is native to the frameworks. If it is native and isn’t hacked on afterward (like below), then you don’t have to muck around in the core code or write a plugin and you can stick to what you know best (which is your application). That is not to say that one should prematurely optimize an application (which is a whole other issue that you need to be careful of) and build it out in a split read/write fashion from the beginning, but that there should be a native way for the application to be faster should you reach that point.

Building this out as a afterthought can be done like what’s below (the example is in Python but can be extrapolated to the language specific to the framework).

You can even do things for the writer in a more transactional fashion using START TRANSACTION and COMMIT if you don’t like using AUTOCOMMIT. You’ll also notice that there is a connect every time a query is executed. A lot of people will have an initial gut reaction of a problem here. In fact, since most of your queries will be taking place over a LAN with a pretty fast backplane (or some other variation of a high speed network), it’s probably negligible. Taking the load off of your master and dispersing it onto slaves will make the most difference here.

All this is a very oversimplified way of taking this step, but it is something that frameworks should consider. Even if its just in a plugin fashion which can be taken advantage of if the database server is getting overloaded.

Related posts:

1. When To Use MySQL Cursor Classes In Python
2. Python Multiprocessing Pools and MySQL
3. Python’s MySQLdb 2014 Error – Commands out of sync

Googling around got me an answer, but I had to read a bunch of different responses to piece together the answer. Essentially this issue (in my case) was a result of the InnoDB engine not loading up when MySQL was restarted. Therefore when MySQL tried to read the frm file (table description) which was written for an InnoDB table with the MyISAM reader, it didn’t like it. Since MyISAM is the fallback engine, it went to that and the table became unusable.

I already knew my table table is an InnoDB table. To be sure that this was the issue, I simply checked to see which engines were loaded (removed some for brevity).

So here I notice that InnoDB is disabled. (Note: I skipped the step where I check my my.cnf to make sure the skip-innodb line in the [mysqld] section was commented out. I already knew it was, but if you are unsure, check.) So I pop over to the error log and I see this:

This says to me that it is likely that the MySQL restart didn’t go as well as the initscript would have liked me to believe. So I see what files are open and what’s running:

Well look at that, 2 versions of MySQL running and ibdata1 is being held open by one of them. So now I do the ugly thing and kill the mysqld process that holding the file lock and then restart MySQL:

So back over to MySQL:

There it is. Now you should be able to start up replication again (if that was the issue). Or if you didn’t discover this issue with replication, you should just be able to use your DB like normal.

Related posts:

1. Python’s MySQLdb 2014 Error – Commands out of sync
2. MySQL Encoded URI Search and Replace
3. Counting Email Addresses By Domain in MySQL

Cassandra is the production database that’s in use on Facebook for handling their email system and Digg.

One thing that I would like to note is that when I tested TC, I used the Perl API for both TC and TT. I tried both the Perl API and the Ruby API. I couldn’t get the Ruby API (written by Evan Weaver of Twitter) it to work at all with the Cassandra server (although I am sure the server included with the gem works well). I initially struggled quite a bit with the UUID aspects of the Perl API until I finally gave up and changed the ColumnFamily CompareWith type from

to

Then everything was working well and I began to write my tests. The layout that I ended up using is going to be one that works in a schemaless fashion. I created 2 consistent columns per user: email and person_id. Here is where it gets interesting and different for those of us used to RDBMS’s and having less columns. For this particular project, every time a user is sent an email, there is a “row” (I call it a row for those unfamiliar with Cassandra terminology, it is actually a column) added in the format of: send_dates_<date> (note the structure below). The value of this column is the mailing campaign id sent to the user on this date. This means that if the user receives 365 emails per year at one a day, then there will be 365 rows (or Cassandra columns) that start with send_dates_ and end with YYYY-MM-DD. Note the data structure below in a JSON ish format.

To understand all the data structures in Cassandra better, I strongly recommend reading WTF Is A SuperColumn Cassandra Data Model and Up And Running With Cassandra. They are written by folks at Digg and Twitter respectively and are well worth the reads.

So for my first iteration, I simply loaded up the data in the format mentioned above. Every insert does an insert of an email and person_id just in case they aren’t there to begin with. The initial data set has approximately 3.6 million records. This caused all sorts of problems with the default configurations (ie kept crashing on me). The changes I made to the default configuration are as follows:

• Change the maximum file descriptors from 1024 (system default) to 65535 (or unlimited)
• Change the default minimum and maximum Java -Xms256M -Xmx2G (could not get the data to load past 2.5 million records without upping max memory values)

It was interesting to note that the write latency of about 3.6 million records was 0.004 ms. Also the data compaction brought the size of the records on disk down from 13G to 1.4G. Those figures are being achieved with the reads and writes happening on the same machines.

The load of the second data set took a mere 30m when compared to loading that same data set into Tokyo Cabinet which took closer to 180m.

Now that there is a dataset worth working with, it’s time to start the read tests.

For the first test, I am doing a simple get of the email column. This is just to iterate over the column and find out the approximate speed of the read operations.

For the second test, I made use of a Cassandra feature called get_slice. Since I have columns that are in the format send_dates_YYYY-MM-DD, I used get_slice to grab all column names on a per-row (each email address) basis that were between send_dates_2009-09-29 and send_dates_2009-10-29. The maximum amount that could be matched were 2 (since I only loaded 2 days worth of data into the data base). I used data from a 3rd day so I can get 0, 1, or 2 as results.

This first run is using the Perl version of the script.

Subsequent runs I began to run into API issues and rewrote the same script in Python to see if the more well tested Thrift Python API was faster than the Thrift Perl API (and wouldn’t give me timeout issues). The Perl timeout issues ended up being fixable, but I continued with the tests in Python.

Ultimately with Cassandra, there was quite a bit of a learning curve. But in my opinion is well worth it. Cassandra is an extremely powerful database system that I plan on continuing to explore in greater detail with a few more in depth tests. If you have the chance, take a look at Cassandra.

Related posts:

1. Tokyo Tyrant and Tokyo Cabinet
2. Counting Frequencies of Frequencies
3. Speeding Up Your Selects and Sorts

Ultimately I am working on a comparison of Cassandra and Tokyo Cabinet, but I will get to more on Cassandra later.

Ideally the tests I am going to be doing are fairly simple. I am going to be loading a few million rows into a TCT database (which is a table database in TC terms) and then loading key, value pairs into the database. The layout in a hash format is basically going to be as follows:

I ran these tests in the following formats for INSERTing the data into the a table database and as serialized data in a hash database. It is necessary to point out that the load on this machine is the normal load. Therefore it cannot be a true benchmark. Since the conditions are not optimal (but really, when are they ever), take the results with a grain of salt. Also, there is some data munging going on during every iteration to grab the email addresses and other data. All this is being done through the Perl API and Tokyo Tyrant. The machine that this is running on is a Dual Dual Core 2.5GHz Intel Xeon processor with 16G of memory.

For the first round, a few things should be noted:

• The totals referenced below are email address counts add/modified in the db
• I am only using 1 connection to the Tokyo Tyrant DB and it is currently setup to handle 8 threads
• I didn’t do any memory adjustment on startup, so the default (which is marginal) is in use
• I am only using the standard put operations, not putcat, putkeep, or putnr (which I will be using later)

The results of the table database are as follows. It is also worth noting the size of the table is around 410M on disk.

The structure for the hash database (seeing as its only key value) is as follows:

The results of loading the same data into a hash database are as follows. It is also worth noting the size of the table is around 360M on disk. This is significantly smaller than the 410M of the table database containing the same style data.

For the second round, I loaded a second days worth of data in to the database. I used the same layouts as above with the following noteworthy items:

• I did a get first prior to the put to decide whether to use put or putcat
• The new data structure is now either “2009-09-30,2009-10-01″ or “2009-09-30|1995,2009-10-01|1996″

Results of the hash database test round 2:

Results of the table database test round 2:

When it comes down to the final implementation, I will likely be parallelizing the put in some form. I would like to think that a database designed for this sort of thing works best in a concurrent environment (especially considering the default startup value is 8 threads).

It is obvious that when it comes to load times, that the hash database is much faster. Now its time to run some queries and see how this stuff goes down.

So I ran some queries first against the table database. I grabbed a new list of 3.6 million email addresses and iterated over the list, grabbed the record from the table database and counted how many dates (via array value counts) were entered for that email address. I ran the script 4 times and results were as follows. I typically throw out the first run since caching kicks in for the other runs.

Doing the same thing for the hash database, I got the following result set:

I think the results speak for themselves. A hash database is obviously faster (which is something most of us assumed from the beginning). The rest of time comes form programmatic comparisons like date comparisons in specific slices of the array. Load times can be sped up using concurrency, but given the requirements of the project, the gets have to be done in this sequential fashion.

Now its on to testing Cassandra in a similar fashion for comparison.

Related posts:

1. Creating a Process Table hash in Perl
2. First Experience With Cassandra
3. MySQL Error 1033: Incorrect Information in File

I want to know which domains make up the majority of the users. (The numbers have been changed to protect the innocent):

mysql> SELECT SUBSTRING_INDEX(email, '@', -1) as Domain, count(*) as Total
      FROM email_list
GROUP BY Domain
ORDER BY Total DESC
       LIMIT 15;
+----------------+---------+
| Domain         | Total   |
+----------------+---------+
| yahoo.com      | 1304000 |
| hotmail.com    |  908400 |
| aol.com        |  800000 |
| msn.com        |  168000 |
| gmail.com      |  161000 |
| comcast.net    |  143000 |
| sbcglobal.net  |  110000 |
| bellsouth.net  |   62000 |
| cox.net        |   58000 |
| verizon.net    |   56000 |
| earthlink.net  |   52000 |
| charter.net    |   46000 |
| juno.com       |   30000 |
| optonline.net  |   22000 |
| netzero.com    |   17000 |
+----------------+---------+

Related posts:

1. List of Feedback Loops
2. Transferring Email From Gmail/Google Apps to Dovecot With Larch
3. SEO and Cross-Domain Content Syndication

What was essentially happening was that the data was being stored as iso-8859-1. The Java code was connecting to the DB in UTF-8 and pulling it into Java (which is usually UTF-16, but in this case was being handled as UTF-8). It was then being sent to the browser as URL encoded UTF-8 when in reality, it hadn’t even properly been put into UTF-8 character encoding. This then gave the web browser some funny yen symbols and question marks. This was not quite what we were aiming for.

The moral of this story is that it is necessary to realize the character encoding of the start point and end point of your data. It is crucial that the code points match up otherwise they could potentially make for an interesting screen given to the reader. All this could have been avoided with a simple: ALTER TABLE myTable MODIFY myColumn VARCHAR(255) CHARACTER SET utf8;.

Related posts:

1. MySQL Encoded URI Search and Replace