I’ve been talking a lot about message systems in distributed architectures lately. And one of the slides I show in my talks is a slide about compressing messages before writes to the database. In other words, if you have 150k messages per second coming in which would translate 1:1 in writes and force your database(s) to incur a 150k write per second load, you pull in all those messages in to memory for a short period (say one minute) and group them and write the group in batch. Depending on how much you can group, you can easily cut your write load by an order of magnitude. (more…)
Batching Work For Efficiency and Tuning
Monday, June 23rd, 2014Pros and Cons of Redis-Resque and SQS
Monday, July 30th, 2012As with any system or application, there are upsides and downsides to using them. The two queueing systems that I want to explore are Resque and Amazon’s Simple Queuing Service. Resque is essentially a set of queuing APIs that run on Redis. Redis is an in-memory data store and is what actually handles the queues. It’s capable of handling complex data structures like lists (what Resque queues use), sets or sorted sets. Amazon’s SQS is an eventually consistent sharded messaging/queueing system.
(more…)
Using Unique Keys and Key Groups with Background Jobs in Gearman::Client
Wednesday, June 16th, 2010While diving into Gearman using Gearman::Client with MySQL and libdrizzle (I know, a mouthful), I ran into what I thought was a bug. I was only able to add 1 background job of any type at a particular time. The launchpad “bug note,” which is available in its entirety here, is rightly labeled won’t fix.
(more…)
Setting Up daemontools on CentOS 5
Friday, March 26th, 2010I recently had to setup daemontools on a CentOS system. I had set it up before but it had been a while. So I Google’d around and found very little and what little I did find wasn’t very helpful. So here is a quick and dirty on setting up daemontools. I even included the CentOS fix that I came across to make it compile. There is also a patch version (if you were building an RPM), but I prefer just making the change in this case; it’s much simpler.
(more…)
Buy My Book
