Erics Tech Blog » MySQL

MySQL for Python

eric — Mon, 27 Dec 2010 07:10:22 +0000

I am always for using the right tool for the right job. A lot of time, that tool is Python. I have always had trouble finding solid documentation on using MySQL with Python. There was generally enough to get by, but the more the merrier. Enter MySQL for Python by Albert Lukaszewski.

MySQL for Python

As I mention in most of my reviews, on the of the things I love about Packt Publishing books is that they typically follow the same pattern: installation, teaching, project, summary. And my favorite piece is the mini projects given at the end of each chapter. It’s a little reality check for the reader reminding them that everything that they are doing has a real life application.

So skipping the installation chapter and jumping right in to the teaching, there is a lot of discussion about how to do the common stuff that one does in MySQL through Python. If you have a lot of experience with MySQL, then this is a handy reference. I also really like the fact that scattered throughout the book is information on becoming a better programmer (like tradeoffs on memory efficiency vs. performance).

Chapter 4 contained information about exception handling. Just like previous chapters, the amount of information on the handling of warnings and exceptions exceeds the boundaries of just MySQL in Python. This was a great review for any Python programmer. No need for further discussion as this chapter is available from Packt Publishing here.

The next few chapters deal with data manipulation in Python. This includes INSERTs, UPDATEs, DELETEs, etc. Chapter 8 specifically deals with user management in MySQL. This is a great skill to have regardless of the interface that you are using to tie into MySQL. Most programmers neglect the concept of administration and leave it to the DBAs or the SysAdmins. Albert takes the reader through some admin exercises including user management, backup and recovery, and accessing the MySQL meta information like information_schema tables. There was also discussion about the various storage engines which was unexpected.

Bouncing backwards a little was a lesson on the string and aggregation functions built into MySQL. A lot of these capabilities are usually handled programatically and not off-loaded to the database like it should be. Again, this is a great refresher (or even initial) lesson for any programmer who spends a lot of time building and maintaining complex queries.

Overall this book was another great teaching tool put forth by the folks at Packt. The audience is definitely not at the beginner Python programmer, but an intermediate level developer would have no issues understanding everything. You can find this book either at Amazon or directly from Packt Publishing.

Database Read/Write Splitting in Frameworks/ORMs

eric — Mon, 01 Mar 2010 13:00:27 +0000

Although one of the primary ideas behind frameworks is to keep things as simple as possible, sometimes they create issues in the long run. What I am about to discuss is something of a luxury problem (as scaling usually is), but it is a problem nonetheless.

When initially starting a project, whether you are using Ruby on Rails (Ruby), Django (Python), CakePHP (PHP), Catalyst (Perl), or any of the other 100s of frameworks in any of the languages out there, the first and most important thing to do is to get it out the door. Once you have done that, it’s time to get users, fix bugs, and add features. After you have done all that and you have a great web app, its time to think scaling. (Yes I realize that I have trivialized this process immensely, but its for a point, I promise).

When starting to scale (whether its out or up) and you decide its time to add another database, its necessary to analyze your app and decide whether its read heavy or write heavy. A lot of scaling comes in knowing your application and where its bottlenecks are. Let’s assume that you are at the point that you need to add a database server. What would be great is if you had a framework that allowed you to set some database servers as read-only in order to take load off the master.

In an abstract format, it would be a good idea to break out your SQL requirements into 2 functions: sql_write_query and sql_read_query. Then have the functions go to your primary database server and slave database servers respectively. The reason that you should do this instead of using a single function that sends the query to the “correct” location based on the SQL it finds is that your slave servers may be behind the master (which is the nature of replication) which could give you an incorrect result in your query. This way, depending on the importance and type of the query, in your code you can choose the location that you want to send the query to. The read queries where accuracy is extremely important can be sent to the database using sql_write_query and all others can be executed normally using your sql_read_query function.

How does this relate to frameworks and ORMs? It would be very handy if frameworks provided a method to expand an application into splitting read and write queries that is native to the frameworks. If it is native and isn’t hacked on afterward (like below), then you don’t have to muck around in the core code or write a plugin and you can stick to what you know best (which is your application). That is not to say that one should prematurely optimize an application (which is a whole other issue that you need to be careful of) and build it out in a split read/write fashion from the beginning, but that there should be a native way for the application to be faster should you reach that point.

Building this out as a afterthought can be done like what’s below (the example is in Python but can be extrapolated to the language specific to the framework).

import MySQLdb

def write_mysql_query(query):
conn = MySQLdb.connect(host = "dbvip1.example.com",
user = "root",
passwd = "pass",
db = "myapp")
cursor = conn.cursor()
cursor.execute("SET AUTOCOMMIT=1")
cursor.execute(query)
cursor.close()
conn.close()

def read_mysql_query(query):
conn = MySQLdb.connect(host = "slavedb1.example.com",
user = "root",
passwd = "pass",
db = "myapp")
cursor = conn.cursor()
cursor.execute(query)
result = cursor.fetchall()
cursor.close()
conn.close()

You can even do things for the writer in a more transactional fashion using START TRANSACTION and COMMIT if you don’t like using AUTOCOMMIT. You’ll also notice that there is a connect every time a query is executed. A lot of people will have an initial gut reaction of a problem here. In fact, since most of your queries will be taking place over a LAN with a pretty fast backplane (or some other variation of a high speed network), it’s probably negligible. Taking the load off of your master and dispersing it onto slaves will make the most difference here.

All this is a very oversimplified way of taking this step, but it is something that frameworks should consider. Even if its just in a plugin fashion which can be taken advantage of if the database server is getting overloaded.

When To Use MySQL Cursor Classes In Python

eric — Mon, 18 Jan 2010 15:00:40 +0000

I have been writing a lot of code that has been interacting with MySQL lately. Sometimes I find it easier to work the result set in a dictionary form and other times it is easier with an array. But in order to not break all your code, it is necessary to set a default cursor class that keeps your code consistent. More often than not, I find using using a arrays is easier since I just want quick access to all the retrieved data. I also end up making my SELECT calls while specifying the columns and order of the columns I want returned.

The reason that using cursor classes is handy is because Python doesn’t come with a mysql_fetch_assoc like PHP or selectrow_hashref like Perl’s DBI interface. Python uses cursor dictionaries to bridge this gap. Ultimately your result is the same. But as with Perl and PHP, defaulting to cursor dictionaries isn’t a good idea for larger datasets because of the extra processing time and memory required to convert the data.

import MySQLdb

conn = MySQLdb.Connect(
host='localhost', user='user',
passwd='secret', db='test')
cursor = conn.cursor()
cursor.execute("SELECT this,that FROM foobar")
rows = cursor.fetchall()
cursor.close()
conn.close()

This then allows me to do just call the DictCursor (dictionary based cursor) on a single cursor object for a particular query and leave all the rest of the queries as array based results.

import MySQLdb
import MySQLdb.cursors

conn = MySQLdb.Connect(
host='localhost', user='user',
passwd='secret', db='test')
cursor = conn.cursor(cursorclass=MySQLdb.cursors.DictCursor)
cursor.execute("SELECT this,that FROM foobar")
rows = cursor.fetchall()
for row in rows:
print row['this'], row['that']
cursor.close()
conn.close()

Note: Doing something like changing the connection string to use a default cursor class of dictionary cursor in an existing code base is not a good idea. It will make your results come back as dictionaries when your current code expects arrays. Yes, I did this and had to track it down the hard way. Hopefully this saves you some trouble.

conn = MySQLdb.Connect(
host='localhost', user='user',
passwd='secret', db='test',
cursorclass=MySQLdb.cursors.DictCursor)

MySQL Error 1033: Incorrect Information in File

eric — Tue, 05 Jan 2010 14:15:43 +0000

If you’ve ever been plagued by an error 1033 issue in MySQL (replication will show it as well), then I might be able to help you out. The error reads something like, “Incorrect information in file: ‘./mydb/table.frm’. I classify this as another one of MySQLs cryptic error messages. Here is how I determined that this was my problem.

Googling around got me an answer, but I had to read a bunch of different responses to piece together the answer. Essentially this issue (in my case) was a result of the InnoDB engine not loading up when MySQL was restarted. Therefore when MySQL tried to read the frm file (table description) which was written for an InnoDB table with the MyISAM reader, it didn’t like it. Since MyISAM is the fallback engine, it went to that and the table became unusable.

Last_Errno: 1033
Last_Error: Error 'Incorrect information in file: './st/table.frm'' on query. Default database: 'mydb'. Query: 'INSERT INTO `table` (`id`,`col1`) VALUES (1,'foobar')'
# or
mysql> REPAIR TABLE table;
+-------------+--------+----------+----------------------------------------------------+
| Table | Op | Msg_type | Msg_text |
+-------------+--------+----------+----------------------------------------------------+
| mydb.table | repair | Error | Incorrect information in file: './mydb/table.frm' |
| mydb.table | repair | error | Corrupt |
+-------------+--------+----------+----------------------------------------------------+
2 rows in set (0.02 sec)

I already knew my table table is an InnoDB table. To be sure that this was the issue, I simply checked to see which engines were loaded (removed some for brevity).

So here I notice that InnoDB is disabled. (Note: I skipped the step where I check my my.cnf to make sure the skip-innodb line in the [mysqld] section was commented out. I already knew it was, but if you are unsure, check.) So I pop over to the error log and I see this:

InnoDB: Unable to lock ./ibdata1, error: 11
InnoDB: Check that you do not already have another mysqld process
InnoDB: using the same InnoDB data or log files.
091222 15:21:55 InnoDB: Unable to open the first data file
InnoDB: Error in opening ./ibdata1
091222 15:21:55 InnoDB: Operating system error number 11 in a file operation.
InnoDB: Error number 11 means 'Resource temporarily unavailable'.
InnoDB: Some operating system error numbers are described at
InnoDB: http://dev.mysql.com/doc/refman/5.0/en/operating-system-error-codes.html
79InnoDB: Could not open or create data files.

This says to me that it is likely that the MySQL restart didn’t go as well as the initscript would have liked me to believe. So I see what files are open and what’s running:

[root@db mysql]# lsof | grep ibdata1
COMMAND PID USER FD TYPE DEVICE SIZE NODE NAME
mysqld 24574 mysql 4uW REG 8,3 5018484736 61538308 /var/lib/mysql/ibdata1
[root@db5 mysql]# ps ax | grep mysqld
24536 pts/0 S 0:00 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql/ --pid-file=/var/lib/mysql//db.example.com.pid
24574 pts/0 Sl 7:58 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql/ --user=mysql --pid-file=/var/lib/mysql//db.example.com.pid --skip-external-locking --port=3306 --socket=/var/run/mysqld/mysqld.sock
26635 pts/0 S 0:00 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql/ --pid-file=/var/lib/mysql//db.example.com.pid
26666 pts/0 Sl 0:06 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql/ --user=mysql --pid-file=/var/lib/mysql//db.example.com.pid --skip-external-locking --port=3306 --socket=/var/run/mysqld/mysqld.sock

Well look at that, 2 versions of MySQL running and ibdata1 is being held open by one of them. So now I do the ugly thing and kill the mysqld process that holding the file lock and then restart MySQL:

[root@db mysql]# kill -9 24574
[root@db mysql]# ps ax | grep mysqld
24536 pts/0 S 0:00 /bin/sh /usr/bin/mysqld_safe --datadir=/var/lib/mysql/ --pid-file=/var/lib/mysql//db.example.com.pid
27051 pts/0 Rl 0:02 /usr/sbin/mysqld --basedir=/ --datadir=/var/lib/mysql/ --user=mysql --pid-file=/var/lib/mysql//db.example.com.pid --skip-external-locking --port=3306 --socket=/var/run/mysqld/mysqld.sock
27075 pts/0 S+ 0:00 grep mysqld
[root@db mysql]# /etc/init.d/mysql restart
MySQL manager or server PID file could not be found! [FAILED]
Starting MySQL............... [ OK ]

So back over to MySQL:

There it is. Now you should be able to start up replication again (if that was the issue). Or if you didn’t discover this issue with replication, you should just be able to use your DB like normal.

Python Multiprocessing Pools and MySQL

eric — Mon, 21 Dec 2009 12:00:00 +0000

There really isn’t a solid Python module for multiprocessing and MySQL. Now this may be because MySQL on a single server is disk bound and therefore limited in speed or just because no one has written it. So here is a quick and dirty example using the Pool module in multiprocessing in Python 2.6 and MySQLdb.

I also tried using PySQLPool. This was designed for threading and not forking as I am doing with Pool method. Although I am sure it is possible to use PySQLPool with forking by passing the connection (pool) object down to the child process or possibly doing something with IPC, I decided to keep it simple (although slightly more expensive) and instantiate MySQLdb connections upon fork.

import MySQLdb
import multiprocessing

def main():
tablename = 'address_demographics'
inserts = []
id = 0
max_id = 1000000
while id < max_id:
sql = """
INSERT IGNORE INTO `%s` (person_id)
SELECT person_id
FROM `people`
WHERE id BETWEEN %d AND %d
""" % (tablename,id,(id + 100000))
inserts.append(sql)
id += 100001
pool = multiprocessing.Pool(multiprocessing.cpu_count())
pool.map(sqlWorkerInsert, inserts, 1)
pool.close()
pool.join()

def sqlWorkerInsert(sql):
conn = MySQLdb.connect(host = "127.0.0.1",
user = "USER",
passwd = "PASS",
db = "DB")
cursor = conn.cursor()
# Need to do this since AUTOCOMMIT = 0 by default (wtf?)
cursor.execute("SET AUTOCOMMIT=1")
cursor.execute(sql)
cursor.close()
conn.close()

It may not be the cleanest method to open a new connection to MySQL for every worker subprocess. Since MySQL opens a new thread for every connection, I don’t think this is that big of a deal. However I am always open to new ways of doing things.

Python’s MySQLdb 2014 Error – Commands out of sync

eric — Fri, 18 Dec 2009 13:00:36 +0000

While writing a simple Python script to access and process data in a database, I came across an error that said:

Error 2014: Commands out of sync; you can't run this command now

After quite a bit of Googling and with very little findings, I had to dive in a little and try to figure out what was going on. The whole error looked like this:

[elubow@web7 scripts]$ ./2014_test.py
Traceback (most recent call last):
File "build/bdist.linux-x86_64/egg/MySQLdb/cursors.py", line 173, in execute
self.errorhandler(self, exc, value)
File "build/bdist.linux-x86_64/egg/MySQLdb/connections.py", line 36, in defaulterrorhandler
raise errorclass, errorvalue
ProgrammingError: (2014, "Commands out of sync; you can't run this command now")
Error 2014: Commands out of sync; you can't run this command now
Exception _mysql_exceptions.ProgrammingError: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'test_table' at line 1") in
> ignored

The issue turned out to be a very simple one to fix. Apparently MySQLdb doesn’t support compound MySQL statements. That is to say that this won’t work:

cursor.execute('SELECT COUNT(*) FROM people; SHOW TABLES LIKE addresses;')

But this will work:

cursor.execute("SELECT COUNT(*) FROM people")
cursor.close()
cursor = conn.cursor()
cursor.execute("SHOW TABLES LIKE addresses")

Make sure you break up your SQL statements into separate execute statements and you should avoid this MySQL 2014 error.

Percona Conference Slides

eric — Sat, 09 May 2009 13:00:44 +0000

For everyone who was not able to attend this year, here is the link to the Percona Perfmance Conference slides: http://conferences.percona.com/percona-performance-conference-2009/schedule.html.

I suggest you check them out if you work with MySQL. Enjoy.

SysAdmin Of The Year Contest

Counting Email Addresses By Domain in MySQL

eric — Wed, 01 Apr 2009 13:50:21 +0000

Every so often I find some statistical need that although Perl program would be easy to write for it, its probably something the database should just handle. So I have a column in an email management table that has just the email addresses in the format user@domain.tld.

I want to know which domains make up the majority of the users. (The numbers have been changed to protect the innocent):

mysql> SELECT SUBSTRING_INDEX(email, '@', -1) as Domain, count(*) as Total
      FROM email_list
GROUP BY Domain
ORDER BY Total DESC
       LIMIT 15;
+----------------+---------+
| Domain         | Total   |
+----------------+---------+
| yahoo.com      | 1304000 |
| hotmail.com    |  908400 |
| aol.com        |  800000 |
| msn.com        |  168000 |
| gmail.com      |  161000 |
| comcast.net    |  143000 |
| sbcglobal.net  |  110000 |
| bellsouth.net  |   62000 |
| cox.net        |   58000 |
| verizon.net    |   56000 |
| earthlink.net  |   52000 |
| charter.net    |   46000 |
| juno.com       |   30000 |
| optonline.net  |   22000 |
| netzero.com    |   17000 |
+----------------+---------+

Character Encoding

eric — Thu, 23 Oct 2008 21:44:58 +0000

I recently ran into some character encoding issues and I wanted to share the fun. The default character encoding for MySQL on Gentoo is latin-1 or iso-8859-1. This wasn’t a problem until we recently started putting content straight from the DB through Java and onto the web. Java connects to the DB with a character encoding (typically UTF-8). Since UTF-8 is roughly a superset of iso-8859-1, it generally wasn’t a problem. Except when UTF-8 and UTF-16 characters were put into an iso-8859-1 database without translation.

What was essentially happening was that the data was being stored as iso-8859-1. The Java code was connecting to the DB in UTF-8 and pulling it into Java (which is usually UTF-16, but in this case was being handled as UTF-8). It was then being sent to the browser as URL encoded UTF-8 when in reality, it hadn’t even properly been put into UTF-8 character encoding. This then gave the web browser some funny yen symbols and question marks. This was not quite what we were aiming for.

The moral of this story is that it is necessary to realize the character encoding of the start point and end point of your data. It is crucial that the code points match up otherwise they could potentially make for an interesting screen given to the reader. All this could have been avoided with a simple: ALTER TABLE myTable MODIFY myColumn VARCHAR(255) CHARACTER SET utf8;.

MySQL Encoded URI Search and Replace