The VARCHAR data type is only variable character size for storage, not for sorting and buffering. In fact, since the MySQL optimizer doesn’t know how big the data in that column can be, it has to allocate the maximum size possible for that column. So for sorting and buffering of the name and email columns below would take up 310 bytes per row.

To fix that you should alter the size of your columns. Imagine having the optimizer have to go through an additional 310 bytes for every row. If there are 500k rows in the table time 310 bytes for each row, that can add up in the amount of memory that the optimizer has to use to perform the sorting/buffering.

Consider the following table of businesses:

You can change the name to column sizes to 75 (name), 50 (email), and 100 (url). But realistically you don’t want to chance things getting cut off, so it may be better off to settle on each column here being a VARCHAR(100). Even that would save drastically on the space required to perform a sort and buffer the results.

Related posts:

1. MySQL Encoded URI Search and Replace
2. Using Unique Keys and Key Groups with Background Jobs in Gearman::Client
3. HOWTO Recreate /dev/null

When initially starting a project, whether you are using Ruby on Rails (Ruby), Django (Python), CakePHP (PHP), Catalyst (Perl), or any of the other 100s of frameworks in any of the languages out there, the first and most important thing to do is to get it out the door. Once you have done that, it’s time to get users, fix bugs, and add features. After you have done all that and you have a great web app, its time to think scaling. (Yes I realize that I have trivialized this process immensely, but its for a point, I promise).

When starting to scale (whether its out or up) and you decide its time to add another database, its necessary to analyze your app and decide whether its read heavy or write heavy. A lot of scaling comes in knowing your application and where its bottlenecks are. Let’s assume that you are at the point that you need to add a database server. What would be great is if you had a framework that allowed you to set some database servers as read-only in order to take load off the master.

In an abstract format, it would be a good idea to break out your SQL requirements into 2 functions: sql_write_query and sql_read_query. Then have the functions go to your primary database server and slave database servers respectively. The reason that you should do this instead of using a single function that sends the query to the “correct” location based on the SQL it finds is that your slave servers may be behind the master (which is the nature of replication) which could give you an incorrect result in your query. This way, depending on the importance and type of the query, in your code you can choose the location that you want to send the query to. The read queries where accuracy is extremely important can be sent to the database using sql_write_query and all others can be executed normally using your sql_read_query function.

How does this relate to frameworks and ORMs? It would be very handy if frameworks provided a method to expand an application into splitting read and write queries that is native to the frameworks. If it is native and isn’t hacked on afterward (like below), then you don’t have to muck around in the core code or write a plugin and you can stick to what you know best (which is your application). That is not to say that one should prematurely optimize an application (which is a whole other issue that you need to be careful of) and build it out in a split read/write fashion from the beginning, but that there should be a native way for the application to be faster should you reach that point.

Building this out as a afterthought can be done like what’s below (the example is in Python but can be extrapolated to the language specific to the framework).

You can even do things for the writer in a more transactional fashion using START TRANSACTION and COMMIT if you don’t like using AUTOCOMMIT. You’ll also notice that there is a connect every time a query is executed. A lot of people will have an initial gut reaction of a problem here. In fact, since most of your queries will be taking place over a LAN with a pretty fast backplane (or some other variation of a high speed network), it’s probably negligible. Taking the load off of your master and dispersing it onto slaves will make the most difference here.

All this is a very oversimplified way of taking this step, but it is something that frameworks should consider. Even if its just in a plugin fashion which can be taken advantage of if the database server is getting overloaded.

Related posts:

1. When To Use MySQL Cursor Classes In Python
2. Python Multiprocessing Pools and MySQL
3. Python’s MySQLdb 2014 Error – Commands out of sync

Googling around got me an answer, but I had to read a bunch of different responses to piece together the answer. Essentially this issue (in my case) was a result of the InnoDB engine not loading up when MySQL was restarted. Therefore when MySQL tried to read the frm file (table description) which was written for an InnoDB table with the MyISAM reader, it didn’t like it. Since MyISAM is the fallback engine, it went to that and the table became unusable.

I already knew my table table is an InnoDB table. To be sure that this was the issue, I simply checked to see which engines were loaded (removed some for brevity).

So here I notice that InnoDB is disabled. (Note: I skipped the step where I check my my.cnf to make sure the skip-innodb line in the [mysqld] section was commented out. I already knew it was, but if you are unsure, check.) So I pop over to the error log and I see this:

This says to me that it is likely that the MySQL restart didn’t go as well as the initscript would have liked me to believe. So I see what files are open and what’s running:

Well look at that, 2 versions of MySQL running and ibdata1 is being held open by one of them. So now I do the ugly thing and kill the mysqld process that holding the file lock and then restart MySQL:

So back over to MySQL:

There it is. Now you should be able to start up replication again (if that was the issue). Or if you didn’t discover this issue with replication, you should just be able to use your DB like normal.

Related posts:

1. Python’s MySQLdb 2014 Error – Commands out of sync
2. MySQL Encoded URI Search and Replace
3. Counting Email Addresses By Domain in MySQL

I want to know which domains make up the majority of the users. (The numbers have been changed to protect the innocent):

mysql> SELECT SUBSTRING_INDEX(email, '@', -1) as Domain, count(*) as Total
      FROM email_list
GROUP BY Domain
ORDER BY Total DESC
       LIMIT 15;
+----------------+---------+
| Domain         | Total   |
+----------------+---------+
| yahoo.com      | 1304000 |
| hotmail.com    |  908400 |
| aol.com        |  800000 |
| msn.com        |  168000 |
| gmail.com      |  161000 |
| comcast.net    |  143000 |
| sbcglobal.net  |  110000 |
| bellsouth.net  |   62000 |
| cox.net        |   58000 |
| verizon.net    |   56000 |
| earthlink.net  |   52000 |
| charter.net    |   46000 |
| juno.com       |   30000 |
| optonline.net  |   22000 |
| netzero.com    |   17000 |
+----------------+---------+

Related posts:

1. List of Feedback Loops
2. Transferring Email From Gmail/Google Apps to Dovecot With Larch
3. SEO and Cross-Domain Content Syndication

What was essentially happening was that the data was being stored as iso-8859-1. The Java code was connecting to the DB in UTF-8 and pulling it into Java (which is usually UTF-16, but in this case was being handled as UTF-8). It was then being sent to the browser as URL encoded UTF-8 when in reality, it hadn’t even properly been put into UTF-8 character encoding. This then gave the web browser some funny yen symbols and question marks. This was not quite what we were aiming for.

The moral of this story is that it is necessary to realize the character encoding of the start point and end point of your data. It is crucial that the code points match up otherwise they could potentially make for an interesting screen given to the reader. All this could have been avoided with a simple: ALTER TABLE myTable MODIFY myColumn VARCHAR(255) CHARACTER SET utf8;.

Related posts:

1. MySQL Encoded URI Search and Replace

To accomplish this, the MySQL REPLACE function was used.

To make our lives easier, let’s create a function from that.

And to wrap it up, let’s use it in an actual query. This query finds all instances of a % and replaces it with its unencoded character equivalent.

Related posts:

1. MySQL Proxy Query Profiling
2. Speeding Up Your Selects and Sorts
3. MySQL Error 1033: Incorrect Information in File