Erics Tech Blog » mysql tricks http://eric.lubow.org Thoughts, musings, and other idealistic (sometimes useful) systems and development hoopla. Fri, 18 Nov 2011 14:56:45 +0000 en hourly 1 http://wordpress.org/?v=3.1.4 Speeding Up Your Selects and Sorts http://eric.lubow.org/2010/databases/mysql/speeding-up-your-selects-and-sorts/ http://eric.lubow.org/2010/databases/mysql/speeding-up-your-selects-and-sorts/#comments Thu, 04 Mar 2010 13:30:07 +0000 eric http://eric.lubow.org/?p=585 When you are using a framework, they typically set your VARCHAR size automatically to 255. This is normally fine since you are letting the framework abstract you away from most of the SQL. But if you interact with your SQL, there is a way to get a decent speed increase on your SELECTs and ORDER BYs when you are working with VARCHARs.

The VARCHAR data type is only variable character size for storage, not for sorting and buffering. In fact, since the MySQL optimizer doesn’t know how big the data in that column can be, it has to allocate the maximum size possible for that column. So for sorting and buffering of the name and email columns below would take up 310 bytes per row.

To fix that you should alter the size of your columns. Imagine having the optimizer have to go through an additional 310 bytes for every row. If there are 500k rows in the table time 310 bytes for each row, that can add up in the amount of memory that the optimizer has to use to perform the sorting/buffering.

Consider the following table of businesses:

mysql> DESC businesses;
+----------------+--------------+------+-----+---------+----------------+
| FIELD          | TYPE         | NULL | KEY | DEFAULT | Extra          |
+----------------+--------------+------+-----+---------+----------------+
| id             | INT(11)      | NO   | PRI | NULL    | AUTO_INCREMENT |
| name           | VARCHAR(255) | YES  |     | NULL    |                |
| url            | VARCHAR(255) | YES  |     | NULL    |                |
| email          | VARCHAR(255) | YES  |     | NULL    |                |
| description    | text         | YES  |     | NULL    |                |
| created_at     | datetime     | YES  |     | NULL    |                |
| updated_at     | datetime     | YES  |     | NULL    |                |
+----------------+--------------+------+-----+---------+----------------+
7 ROWS IN SET (0.00 sec)

mysql> SELECT MAX(LENGTH(name)), MAX(LENGTH(email)), MAX(LENGTH(url)) FROM businesses;
+-------------------+--------------------+------------------+
| MAX(LENGTH(name)) | MAX(LENGTH(email)) | MAX(LENGTH(url)) |
+-------------------+--------------------+------------------+
|                53 |                 36 |               40 |
+-------------------+--------------------+------------------+
1 ROW IN SET (0.40 sec)

You can change the name to column sizes to 75 (name), 50 (email), and 100 (url). But realistically you don’t want to chance things getting cut off, so it may be better off to settle on each column here being a VARCHAR(100). Even that would save drastically on the space required to perform a sort and buffer the results.

Related posts:

  1. MySQL Encoded URI Search and Replace
  2. Using Unique Keys and Key Groups with Background Jobs in Gearman::Client
  3. HOWTO Recreate /dev/null

]]> http://eric.lubow.org/2010/databases/mysql/speeding-up-your-selects-and-sorts/feed/ 3 MySQL Encoded URI Search and Replace http://eric.lubow.org/2008/databases/mysql/mysql-encoded-uri-search-and-replace/ http://eric.lubow.org/2008/databases/mysql/mysql-encoded-uri-search-and-replace/#comments Fri, 26 Sep 2008 14:00:53 +0000 eric http://eric.lubow.org/?p=73 So I can definitely not take credit for this trick. The credit goes to Augusto Bott of the Pythian Group for this. I have a table in my DB that has columns of encoded URIs. The list of URI encoding character translations is available here.

To accomplish this, the MySQL REPLACE function was used.

mysql> SET @test:='%20%27%7C%26%5E%2B%2D%25';
Query OK, 0 rows affected (0.00 sec)

mysql> SELECT @test as original,
    ->        REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(
    ->        @test,'%20',' '),
    ->        '%27','\''),
    ->        '%7C','|'),  -- REPLACE() is case sensitive
    ->        '%7c','|'),  -- so we have
    ->        '%26','&'),
    ->        '%5E','^'),
    ->        '%5e','^'),  -- to repeat
    ->        '%2D','-'),
    ->        '%2d','-'),  -- ourselves
    ->        '%2B','+'),
    ->        '%2b','+'),  -- sometimes
    ->        '%25','%') as replaced;
+--------------------------+----------+
| original                 | replaced |
+--------------------------+----------+
| %20%27%7C%26%5E%2B%2D%25 |  '|&^+-% |
+--------------------------+----------+
1 row in set (0.01 sec)

To make our lives easier, let’s create a function from that.

mysql> CREATE FUNCTION url_replace
    -> (url VARCHAR(1024))
    -> RETURNS VARCHAR(1025)
    -> BEGIN
    -> RETURN REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(
    ->        @test,'%20',' '),
    ->        '%27','\''),
    ->        '%7C','|'),
    ->        '%7c','|'),
    ->        '%26','&'),
    ->        '%5E','^'),
    ->        '%5e','^'),
    ->        '%2D','-'),
    ->        '%2d','-'),
    ->        '%2B','+'),
    ->        '%2b','+'),
    ->        '%25','%');
    -> END$$
Query OK, 0 rows affected (0.02 sec)

And to wrap it up, let’s use it in an actual query. This query finds all instances of a % and replaces it with its unencoded character equivalent.

mysql> UPDATE website_page SET uri = REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE(REPLACE( uri,'%20',' '), '%27','\''), '%7C','|'), '%7c','|'), '%26','&'), '%5E','^'), '%5e','^'), '%2D','-'), '%2d','-'), '%2B','+'), '%2b','+'), '%25','%'), '%2C',',') WHERE uri LIKE '%\%%';

Related posts:

  1. MySQL Proxy Query Profiling
  2. Speeding Up Your Selects and Sorts
  3. MySQL Error 1033: Incorrect Information in File

]]> http://eric.lubow.org/2008/databases/mysql/mysql-encoded-uri-search-and-replace/feed/ 0