Erics Tech Blog » python

MySQL for Python

eric — Mon, 27 Dec 2010 07:10:22 +0000

I am always for using the right tool for the right job. A lot of time, that tool is Python. I have always had trouble finding solid documentation on using MySQL with Python. There was generally enough to get by, but the more the merrier. Enter MySQL for Python by Albert Lukaszewski.

MySQL for Python

As I mention in most of my reviews, on the of the things I love about Packt Publishing books is that they typically follow the same pattern: installation, teaching, project, summary. And my favorite piece is the mini projects given at the end of each chapter. It’s a little reality check for the reader reminding them that everything that they are doing has a real life application.

So skipping the installation chapter and jumping right in to the teaching, there is a lot of discussion about how to do the common stuff that one does in MySQL through Python. If you have a lot of experience with MySQL, then this is a handy reference. I also really like the fact that scattered throughout the book is information on becoming a better programmer (like tradeoffs on memory efficiency vs. performance).

Chapter 4 contained information about exception handling. Just like previous chapters, the amount of information on the handling of warnings and exceptions exceeds the boundaries of just MySQL in Python. This was a great review for any Python programmer. No need for further discussion as this chapter is available from Packt Publishing here.

The next few chapters deal with data manipulation in Python. This includes INSERTs, UPDATEs, DELETEs, etc. Chapter 8 specifically deals with user management in MySQL. This is a great skill to have regardless of the interface that you are using to tie into MySQL. Most programmers neglect the concept of administration and leave it to the DBAs or the SysAdmins. Albert takes the reader through some admin exercises including user management, backup and recovery, and accessing the MySQL meta information like information_schema tables. There was also discussion about the various storage engines which was unexpected.

Bouncing backwards a little was a lesson on the string and aggregation functions built into MySQL. A lot of these capabilities are usually handled programatically and not off-loaded to the database like it should be. Again, this is a great refresher (or even initial) lesson for any programmer who spends a lot of time building and maintaining complex queries.

Overall this book was another great teaching tool put forth by the folks at Packt. The audience is definitely not at the beginner Python programmer, but an intermediate level developer would have no issues understanding everything. You can find this book either at Amazon or directly from Packt Publishing.

Philosophical Python

eric — Fri, 19 Feb 2010 13:30:43 +0000

Having spent a lot of time in the Perl world and even trying to write a Perl poem here and there, I decided to give some Python poetry a try. It turned out to be a bit more philosophical than poetic. But it compiles and that’s what counts; even though it doesn’t do anything useful (or anything at all).

Since as people, we get the most out of something by filling in small spaces with our imagination (think of how comic books are so successful), fill in the words that might be missing. Either way, enjoy. Hope you get as much from reading it as I did from writing it.

def people():
try:
class Me(being):
def __init__(self):
self.mind = MyThoughts
self.body = MyBeing
self.soul = MyEssence

def inner(self):
answers = None
answers.seek()
mind_answers = self.mind.read()
body_answers = self.body.read()
soul_answers = self.soul.read()
answers = reduce(mind_answers, body_answers, soul_answers)
if MeaningOfLife not in answers:
raise Awareness, "Searching for meaning"
answers = filter(Intelligence, mind_answers, body_answers, soul_answers)
answers.sort()
return answers

except Awareness, MeaningOfLife:
return MeaningOfLife

When To Use MySQL Cursor Classes In Python

eric — Mon, 18 Jan 2010 15:00:40 +0000

I have been writing a lot of code that has been interacting with MySQL lately. Sometimes I find it easier to work the result set in a dictionary form and other times it is easier with an array. But in order to not break all your code, it is necessary to set a default cursor class that keeps your code consistent. More often than not, I find using using a arrays is easier since I just want quick access to all the retrieved data. I also end up making my SELECT calls while specifying the columns and order of the columns I want returned.

The reason that using cursor classes is handy is because Python doesn’t come with a mysql_fetch_assoc like PHP or selectrow_hashref like Perl’s DBI interface. Python uses cursor dictionaries to bridge this gap. Ultimately your result is the same. But as with Perl and PHP, defaulting to cursor dictionaries isn’t a good idea for larger datasets because of the extra processing time and memory required to convert the data.

import MySQLdb

conn = MySQLdb.Connect(
host='localhost', user='user',
passwd='secret', db='test')
cursor = conn.cursor()
cursor.execute("SELECT this,that FROM foobar")
rows = cursor.fetchall()
cursor.close()
conn.close()

This then allows me to do just call the DictCursor (dictionary based cursor) on a single cursor object for a particular query and leave all the rest of the queries as array based results.

import MySQLdb
import MySQLdb.cursors

conn = MySQLdb.Connect(
host='localhost', user='user',
passwd='secret', db='test')
cursor = conn.cursor(cursorclass=MySQLdb.cursors.DictCursor)
cursor.execute("SELECT this,that FROM foobar")
rows = cursor.fetchall()
for row in rows:
print row['this'], row['that']
cursor.close()
conn.close()

Note: Doing something like changing the connection string to use a default cursor class of dictionary cursor in an existing code base is not a good idea. It will make your results come back as dictionaries when your current code expects arrays. Yes, I did this and had to track it down the hard way. Hopefully this saves you some trouble.

conn = MySQLdb.Connect(
host='localhost', user='user',
passwd='secret', db='test',
cursorclass=MySQLdb.cursors.DictCursor)

Python Multiprocessing Pools and MySQL

eric — Mon, 21 Dec 2009 12:00:00 +0000

There really isn’t a solid Python module for multiprocessing and MySQL. Now this may be because MySQL on a single server is disk bound and therefore limited in speed or just because no one has written it. So here is a quick and dirty example using the Pool module in multiprocessing in Python 2.6 and MySQLdb.

I also tried using PySQLPool. This was designed for threading and not forking as I am doing with Pool method. Although I am sure it is possible to use PySQLPool with forking by passing the connection (pool) object down to the child process or possibly doing something with IPC, I decided to keep it simple (although slightly more expensive) and instantiate MySQLdb connections upon fork.

import MySQLdb
import multiprocessing

def main():
tablename = 'address_demographics'
inserts = []
id = 0
max_id = 1000000
while id < max_id:
sql = """
INSERT IGNORE INTO `%s` (person_id)
SELECT person_id
FROM `people`
WHERE id BETWEEN %d AND %d
""" % (tablename,id,(id + 100000))
inserts.append(sql)
id += 100001
pool = multiprocessing.Pool(multiprocessing.cpu_count())
pool.map(sqlWorkerInsert, inserts, 1)
pool.close()
pool.join()

def sqlWorkerInsert(sql):
conn = MySQLdb.connect(host = "127.0.0.1",
user = "USER",
passwd = "PASS",
db = "DB")
cursor = conn.cursor()
# Need to do this since AUTOCOMMIT = 0 by default (wtf?)
cursor.execute("SET AUTOCOMMIT=1")
cursor.execute(sql)
cursor.close()
conn.close()

It may not be the cleanest method to open a new connection to MySQL for every worker subprocess. Since MySQL opens a new thread for every connection, I don’t think this is that big of a deal. However I am always open to new ways of doing things.

Python’s MySQLdb 2014 Error – Commands out of sync

eric — Fri, 18 Dec 2009 13:00:36 +0000

While writing a simple Python script to access and process data in a database, I came across an error that said:

Error 2014: Commands out of sync; you can't run this command now

After quite a bit of Googling and with very little findings, I had to dive in a little and try to figure out what was going on. The whole error looked like this:

[elubow@web7 scripts]$ ./2014_test.py
Traceback (most recent call last):
File "build/bdist.linux-x86_64/egg/MySQLdb/cursors.py", line 173, in execute
self.errorhandler(self, exc, value)
File "build/bdist.linux-x86_64/egg/MySQLdb/connections.py", line 36, in defaulterrorhandler
raise errorclass, errorvalue
ProgrammingError: (2014, "Commands out of sync; you can't run this command now")
Error 2014: Commands out of sync; you can't run this command now
Exception _mysql_exceptions.ProgrammingError: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'test_table' at line 1") in
> ignored

The issue turned out to be a very simple one to fix. Apparently MySQLdb doesn’t support compound MySQL statements. That is to say that this won’t work:

cursor.execute('SELECT COUNT(*) FROM people; SHOW TABLES LIKE addresses;')

But this will work:

cursor.execute("SELECT COUNT(*) FROM people")
cursor.close()
cursor = conn.cursor()
cursor.execute("SHOW TABLES LIKE addresses")

Make sure you break up your SQL statements into separate execute statements and you should avoid this MySQL 2014 error.

SSH Over The Web With Web Shell

eric — Fri, 27 Nov 2009 11:45:46 +0000

After reading a Tweet from Matt Cutts about being able to SSH from the iPhone (and the web in general), I had to give it a try. I am always looking for better ways to be able to check on systems when necessary. I have iPhone apps for SSHing around if I need as well, but like with any “new” tool, I have to try it out to see if it serves a purpose or makes my admin life easier in any way.

First go check out the Google Code repository for Web Shell. Webshell is written in Python and is based on Ajaxterm. All that’s required is SSL And Python 2.3 or greater. It works on any browser that has Javascript and can make use of AJAX.

The way Web Shell works is you start it up on a server and then can use a web browser to access only that machine over SSH. The works best if you have a gateway server to a network and use a single point of entry to access the rest of the servers. Web Shell runs on HTTPS on port 8022. Reading the README will lead you through the same set of instructions I used below. Once installed, we connect by using a web browser: https://server.com:8022/

elubow@server:/home/web/webshell.lubow.org-8022$ ./make_certificate.sh
Generating a 1024 bit RSA private key
...............................................................++++++
...........................................++++++
writing new private key to 'webshell.pem'
-----
elubow@server:/home/web/webshell.lubow.org-8022$ ./webshell.py
The python SSL extensions seem to be not installed.
You can run WebShell without SSL encryption with the --ssl-disable command line switch.

You’ll notice that while it is possible to run the terminal without SSL, it is just not in our best interest. So let’s go ahead and install it. I use Debian on my servers. So in my case, it’s as simple as:

server:~# apt-get install python-pyopenssl python-openssl
Get:1 http://ftp.de.debian.org lenny/main python-openssl 0.7-2 [88.9kB]
Get:2 http://ftp.de.debian.org lenny/main python-pyopenssl 0.7-2 [9.0kB]
Unpacking python-openssl (from .../python-openssl_0.7-2_i386.deb) ...
Setting up python-openssl (0.7-2) ...
Setting up python-pyopenssl (0.7-2) ...
Processing triggers for python-support ...

Then we run the server:

elubow@server:/home/web/ssh.lubow.org-8022$ ./webshell.py
WebShell (https) at 0.0.0.0, port 8022

I’ve included a screenshot from my iPhone below. The background is interchangeable with any other transparent background image. If you’re in need of a quick SSH or someone else’s smart phone has service and yours doesn’t with your fancy SSH programs, then this is definitely a cool and viable alternative.

Web Shell SSH on iPhone

Web Shell SSH with keyboard out

MySQL for Python