Using Unique Keys and Key Groups with Background Jobs in Gearman::Client

While diving into Gearman using Gearman::Client with MySQL and libdrizzle (I know, a mouthful), I ran into what I thought was a bug. I was only able to add 1 background job of any type at a particular time. The launchpad “bug note,” which is available in its entirety here, is rightly labeled won’t fix.

Tracking that post down wasn’t too difficult, but it would have saved me time if that information was in the Perl documentation. Using Gearman::Client you can create your unique ID (or UUID). To create a background job, your code would then look like this:

use Gearman::Client;
use UUID::Tiny ':std';

my $client = Gearman::Client->new;
$client->dispatch_background("send_email", $json,
   {  
     on_complete => sub { print ${ $_[0] }, "\n" },
     uniq => create_uuid_as_string(),                                                            
   }  
 );

This will ensure that you have a unique UUID when INSERTing the job. But I think that we can do better and this is where the good practice comes into effect (and why the Gearman devs marked it as won’t fix). So let’s think a little more abstractly for a moment. For the uniq key (unique_key column), how about something like “${type}_”.create_uuid_as_string(). This will allow us to group (in this case) our mailing types so that we can monitor the status of our work queue. Yes, we can easily hop into the database and run this query:

mysql> SELECT function_name,COUNT(function_name) FROM gearman_queue GROUP BY function_name;
+---------------+----------------------+
| function_name | COUNT(function_name) |
+---------------+----------------------+
| send_email    |                    3 |
+---------------+----------------------+

But wouldn’t it be nicer to know exactly where we are at with this query:

mysql> SELECT IFNULL(function_name, 'Total') AS FUNCTION, SUBSTRING_INDEX(unique_key, '_', 1) AS mail_type, COUNT(*) AS Total
FROM gearman_queue
GROUP BY function_name, mail_type
WITH ROLLUP;
+------------+-----------+-------+
| FUNCTION   | mail_type | Total |
+------------+-----------+-------+
| send_email | passreset |     2 |
| send_email | regemail  |     1 |
| send_email | NULL      |     3 |
| Total      | NULL      |     3 |
+------------+-----------+-------+

For reference, here are the unique_key columns that allowed to generate that query:

mysql> SELECT unique_key FROM gearman_queue;
+------------------------------------------------+
| unique_key                                     |
+------------------------------------------------+
| passreset_1cdb347b-788f-11df-ad7d-b190002f5dcd |
| passreset_21ed3ae5-788f-11df-8395-aed7743cedde |
| regemail_2a40bebe-788f-11df-bda0-8fb882f63de5  |
+------------------------------------------------+
  • Manuel Müller

    This would have saved me a lot of time. I do not understand the reasons in not auto generating a unique number for each job – or at least making it optional .. but you barely see this while trying to set up gearman.