Erics Tech Blog » python

Philosophical Python

eric — Fri, 19 Feb 2010 13:30:43 +0000

Having spent a lot of time in the Perl world and even trying to write a Perl poem here and there, I decided to give some Python poetry a try. It turned out to be a bit more philosophical than poetic. But it compiles and that’s what counts; even though it doesn’t do anything useful (or anything at all).

Since as people, we get the most out of something by filling in small spaces with our imagination (think of how comic books are so successful), fill in the words that might be missing. Either way, enjoy. Hope you get as much from reading it as I did from writing it.

def people():
try:
class Me(being):
def __init__(self):
self.mind = MyThoughts
self.body = MyBeing
self.soul = MyEssence

def inner(self):
answers = None
answers.seek()
mind_answers = self.mind.read()
body_answers = self.body.read()
soul_answers = self.soul.read()
answers = reduce(mind_answers, body_answers, soul_answers)
if MeaningOfLife not in answers:
raise Awareness, "Searching for meaning"
answers = filter(Intelligence, mind_answers, body_answers, soul_answers)
answers.sort()
return answers

except Awareness, MeaningOfLife:
return MeaningOfLife

When To Use MySQL Cursor Classes In Python

eric — Mon, 18 Jan 2010 15:00:40 +0000

I have been writing a lot of code that has been interacting with MySQL lately. Sometimes I find it easier to work the result set in a dictionary form and other times it is easier with an array. But in order to not break all your code, it is necessary to set a default cursor class that keeps your code consistent. More often than not, I find using using a arrays is easier since I just want quick access to all the retrieved data. I also end up making my SELECT calls while specifying the columns and order of the columns I want returned.

The reason that using cursor classes is handy is because Python doesn’t come with a mysql_fetch_assoc like PHP or selectrow_hashref like Perl’s DBI interface. Python uses cursor dictionaries to bridge this gap. Ultimately your result is the same. But as with Perl and PHP, defaulting to cursor dictionaries isn’t a good idea for larger datasets because of the extra processing time and memory required to convert the data.

import MySQLdb

conn = MySQLdb.Connect(
host='localhost', user='user',
passwd='secret', db='test')
cursor = conn.cursor()
cursor.execute("SELECT this,that FROM foobar")
rows = cursor.fetchall()
cursor.close()
conn.close()

This then allows me to do just call the DictCursor (dictionary based cursor) on a single cursor object for a particular query and leave all the rest of the queries as array based results.

import MySQLdb
import MySQLdb.cursors

conn = MySQLdb.Connect(
host='localhost', user='user',
passwd='secret', db='test')
cursor = conn.cursor(cursorclass=MySQLdb.cursors.DictCursor)
cursor.execute("SELECT this,that FROM foobar")
rows = cursor.fetchall()
for row in rows:
print row['this'], row['that']
cursor.close()
conn.close()

Note: Doing something like changing the connection string to use a default cursor class of dictionary cursor in an existing code base is not a good idea. It will make your results come back as dictionaries when your current code expects arrays. Yes, I did this and had to track it down the hard way. Hopefully this saves you some trouble.

conn = MySQLdb.Connect(
host='localhost', user='user',
passwd='secret', db='test',
cursorclass=MySQLdb.cursors.DictCursor)

Python Multiprocessing Pools and MySQL

eric — Mon, 21 Dec 2009 12:00:00 +0000

There really isn’t a solid Python module for multiprocessing and MySQL. Now this may be because MySQL on a single server is disk bound and therefore limited in speed or just because no one has written it. So here is a quick and dirty example using the Pool module in multiprocessing in Python 2.6 and MySQLdb.

I also tried using PySQLPool. This was designed for threading and not forking as I am doing with Pool method. Although I am sure it is possible to use PySQLPool with forking by passing the connection (pool) object down to the child process or possibly doing something with IPC, I decided to keep it simple (although slightly more expensive) and instantiate MySQLdb connections upon fork.

import MySQLdb
import multiprocessing

def main():
tablename = 'address_demographics'
inserts = []
id = 0
max_id = 1000000
while id < max_id:
sql = """
INSERT IGNORE INTO `%s` (person_id)
SELECT person_id
FROM `people`
WHERE id BETWEEN %d AND %d
""" % (tablename,id,(id + 100000))
inserts.append(sql)
id += 100001
pool = multiprocessing.Pool(multiprocessing.cpu_count())
pool.map(sqlWorkerInsert, inserts, 1)
pool.close()
pool.join()

def sqlWorkerInsert(sql):
conn = MySQLdb.connect(host = "127.0.0.1",
user = "USER",
passwd = "PASS",
db = "DB")
cursor = conn.cursor()
# Need to do this since AUTOCOMMIT = 0 by default (wtf?)
cursor.execute("SET AUTOCOMMIT=1")
cursor.execute(sql)
cursor.close()
conn.close()

It may not be the cleanest method to open a new connection to MySQL for every worker subprocess. Since MySQL opens a new thread for every connection, I don’t think this is that big of a deal. However I am always open to new ways of doing things.

Python’s MySQLdb 2014 Error – Commands out of sync

eric — Fri, 18 Dec 2009 13:00:36 +0000

While writing a simple Python script to access and process data in a database, I came across an error that said:

Error 2014: Commands out of sync; you can't run this command now

After quite a bit of Googling and with very little findings, I had to dive in a little and try to figure out what was going on. The whole error looked like this:

[elubow@web7 scripts]$ ./2014_test.py
Traceback (most recent call last):
File "build/bdist.linux-x86_64/egg/MySQLdb/cursors.py", line 173, in execute
self.errorhandler(self, exc, value)
File "build/bdist.linux-x86_64/egg/MySQLdb/connections.py", line 36, in defaulterrorhandler
raise errorclass, errorvalue
ProgrammingError: (2014, "Commands out of sync; you can't run this command now")
Error 2014: Commands out of sync; you can't run this command now
Exception _mysql_exceptions.ProgrammingError: (1064, "You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near 'test_table' at line 1") in
> ignored

The issue turned out to be a very simple one to fix. Apparently MySQLdb doesn’t support compound MySQL statements. That is to say that this won’t work:

cursor.execute('SELECT COUNT(*) FROM people; SHOW TABLES LIKE addresses;')

But this will work:

cursor.execute("SELECT COUNT(*) FROM people")
cursor.close()
cursor = conn.cursor()
cursor.execute("SHOW TABLES LIKE addresses")

Make sure you break up your SQL statements into separate execute statements and you should avoid this MySQL 2014 error.