Eric's Tech Blog

Changing Shoes For A Redesign

eric — Tue, 09 Mar 2010 11:45:49 +0000

The best way to rethink things is to be in the shoes of your users. Use your app how they use your app. Try to take a fresh look at your application like you’ve never seen it before. Would you change the location of the menu/navigation? Would you change the actual menus/navigation? Would you add a shortcut search box where there wasn’t one before? Maybe you remove the advertising or move the place that the ads are located so that they are less intrusive…

The idea is that every so often you need to take a step back. Looking at your application from your users perspective may well change how your entire application works. I’m not saying this from a statistical analysis of the way people click and heatmaps and all that good stuff (though they do have their applications), I’m saying just a pure usability test from another perspective. Where do the new users look? Where do they click? What’s the first thing they want to go to? Are you putting them through information overload?

So take a step back, change shoes and take a fresh look at your app. No statistics, no heatmaps, no preconceived notions about the problem you are trying to solve (I know this is easier said than done). Just remember why you wrote your app in the first place. Try the passion on for size again and see if that doesn’t stir things up a bit.

No related posts.

Speeding Up Your Selects and Sorts

eric — Thu, 04 Mar 2010 13:30:07 +0000

When you are using a framework, they typically set your VARCHAR size automatically to 255. This is normally fine since you are letting the framework abstract you away from most of the SQL. But if you interact with your SQL, there is a way to get a decent speed increase on your SELECTs and ORDER BYs when you are working with VARCHARs.

The VARCHAR data type is only variable character size for storage, not for sorting and buffering. In fact, since the MySQL optimizer doesn’t know how big the data in that column can be, it has to allocate the maximum size possible for that column. So for sorting and buffering of the name and email columns below would take up 310 bytes per row.

To fix that you should alter the size of your columns. Imagine having the optimizer have to go through an additional 310 bytes for every row. If there are 500k rows in the table time 310 bytes for each row, that can add up in the amount of memory that the optimizer has to use to perform the sorting/buffering.

Consider the following table of businesses:

You can change the name to column sizes to 75 (name), 50 (email), and 100 (url). But realistically you don’t want to chance things getting cut off, so it may be better off to settle on each column here being a VARCHAR(100). Even that would save drastically on the space required to perform a sort and buffer the results.

Database Read/Write Splitting in Frameworks/ORMs

eric — Mon, 01 Mar 2010 13:00:27 +0000

Although one of the primary ideas behind frameworks is to keep things as simple as possible, sometimes they create issues in the long run. What I am about to discuss is something of a luxury problem (as scaling usually is), but it is a problem nonetheless.

When initially starting a project, whether you are using Ruby on Rails (Ruby), Django (Python), CakePHP (PHP), Catalyst (Perl), or any of the other 100s of frameworks in any of the languages out there, the first and most important thing to do is to get it out the door. Once you have done that, it’s time to get users, fix bugs, and add features. After you have done all that and you have a great web app, its time to think scaling. (Yes I realize that I have trivialized this process immensely, but its for a point, I promise).

When starting to scale (whether its out or up) and you decide its time to add another database, its necessary to analyze your app and decide whether its read heavy or write heavy. A lot of scaling comes in knowing your application and where its bottlenecks are. Let’s assume that you are at the point that you need to add a database server. What would be great is if you had a framework that allowed you to set some database servers as read-only in order to take load off the master.

In an abstract format, it would be a good idea to break out your SQL requirements into 2 functions: sql_write_query and sql_read_query. Then have the functions go to your primary database server and slave database servers respectively. The reason that you should do this instead of using a single function that sends the query to the “correct” location based on the SQL it finds is that your slave servers may be behind the master (which is the nature of replication) which could give you an incorrect result in your query. This way, depending on the importance and type of the query, in your code you can choose the location that you want to send the query to. The read queries where accuracy is extremely important can be sent to the database using sql_write_query and all others can be executed normally using your sql_read_query function.

How does this relate to frameworks and ORMs? It would be very handy if frameworks provided a method to expand an application into splitting read and write queries that is native to the frameworks. If it is native and isn’t hacked on afterward (like below), then you don’t have to muck around in the core code or write a plugin and you can stick to what you know best (which is your application). That is not to say that one should prematurely optimize an application (which is a whole other issue that you need to be careful of) and build it out in a split read/write fashion from the beginning, but that there should be a native way for the application to be faster should you reach that point.

Building this out as a afterthought can be done like what’s below (the example is in Python but can be extrapolated to the language specific to the framework).

import MySQLdb

def write_mysql_query(query):
conn = MySQLdb.connect(host = "dbvip1.example.com",
user = "root",
passwd = "pass",
db = "myapp")
cursor = conn.cursor()
cursor.execute("SET AUTOCOMMIT=1")
cursor.execute(query)
cursor.close()
conn.close()

def read_mysql_query(query):
conn = MySQLdb.connect(host = "slavedb1.example.com",
user = "root",
passwd = "pass",
db = "myapp")
cursor = conn.cursor()
cursor.execute(query)
result = cursor.fetchall()
cursor.close()
conn.close()

You can even do things for the writer in a more transactional fashion using START TRANSACTION and COMMIT if you don’t like using AUTOCOMMIT. You’ll also notice that there is a connect every time a query is executed. A lot of people will have an initial gut reaction of a problem here. In fact, since most of your queries will be taking place over a LAN with a pretty fast backplane (or some other variation of a high speed network), it’s probably negligible. Taking the load off of your master and dispersing it onto slaves will make the most difference here.

All this is a very oversimplified way of taking this step, but it is something that frameworks should consider. Even if its just in a plugin fashion which can be taken advantage of if the database server is getting overloaded.

Textmate Minimap Plugin

eric — Thu, 25 Feb 2010 12:00:35 +0000

My editor of choice for most of what I do on the Mac (when I’m not in the shell using Vim) is Textmate. I recently came across a neat plugin called the Textmate Minimap. Essentially this plugin opens a map across the side of your Textmate window (like another drawer on the other side) and allows you to scroll full screens. You can move the editing window screen by screen in the minimap. This is equivalent to a visual page down.

Example of the minimap

This isn’t really handy if you have a lot of small files, but you can imagine the utility with larger files. Now its still in beta stages, so be weary when you try it out. Install it into Textmate as a bundle and use it as necessary.

Philosophical Python

eric — Fri, 19 Feb 2010 13:30:43 +0000

Having spent a lot of time in the Perl world and even trying to write a Perl poem here and there, I decided to give some Python poetry a try. It turned out to be a bit more philosophical than poetic. But it compiles and that’s what counts; even though it doesn’t do anything useful (or anything at all).

Since as people, we get the most out of something by filling in small spaces with our imagination (think of how comic books are so successful), fill in the words that might be missing. Either way, enjoy. Hope you get as much from reading it as I did from writing it.

def people():
try:
class Me(being):
def __init__(self):
self.mind = MyThoughts
self.body = MyBeing
self.soul = MyEssence

def inner(self):
answers = None
answers.seek()
mind_answers = self.mind.read()
body_answers = self.body.read()
soul_answers = self.soul.read()
answers = reduce(mind_answers, body_answers, soul_answers)
if MeaningOfLife not in answers:
raise Awareness, "Searching for meaning"
answers = filter(Intelligence, mind_answers, body_answers, soul_answers)
answers.sort()
return answers

except Awareness, MeaningOfLife:
return MeaningOfLife

Sortfix Or A Next Generation Search Tool

eric — Tue, 16 Feb 2010 14:45:25 +0000

Although SortFix has been around for a while, I hadn’t heard of it until recently. I also didn’t really have a reason to give it a try because the vast majority of my Google searches were providing me with the results I was looking for.

The idea here is that you can do a normal search for something and then you will be presented with a screen that will allow you to narrow down your search using a graphical user interface (GUI). This will produce a search similar to something that a power searcher would do in Google. After doing this a few times, it becomes fairly evident how its done and one could start “power searching” on their own.

Now what’s exciting to me here is the potential. SortFix currently only uses Google as a backend search engine. There is nothing wrong with that considering that most of us only stick to one search engine anyway. However it would be great if SortFix could act as a meta search engine (a search engine that searches multiple other search engines) to do power searches and search the major players using your search terms, pair down the dupes and give you the results. Technically speaking, I am aware that this is no trivial task. But a unified search engine like that would certainly help quite a bit with finding what you’re looking for in a difficult search.

If you haven’t seen SortFix, then I’d head on over there. It’s a concept with some great potential and a cool user friendly searching style.

Social Search’s Effect on SEO

eric — Fri, 12 Feb 2010 16:36:50 +0000

There always seems to be something affecting your SEO rankings. Most recently, people have had issues with personalized search. Another kink in the chain may be social search.

The first thing is that Social Search can’t replace Hyptertext search (such as Google/Yahoo!/Bing/etc). Social search has 3 inherent drawbacks, time to receive information, credibility of the source, and subject/objectivity. Technically subjectivity is an issue with hypertext searches as well, but since the information is considered more permanent, people tend to be a little less opinionated and a little more objective (if the information is factual).

According to a recent NY Times article, the majority of social searches are answered within 10 minutes. In our culture, 10 minutes is a long time to wait. We are all about instant gratification. With regard to credibility, most of the time social searches are requests for opinions, so credibility isn’t an issue. What one thinks of a restaurant or some other establishment is generally an opinion based question.

Ultimately I don’t believe there will be a direct impact on SEO. Social search just aim to fill a different niche. People have a different goal in mind when performing a social search than when they are performing a non-social (hypertext) search.

The one thing that this article in the times neglected to mention is that a lot of the hypertext search results are becoming social. With the hypertext search engines indexing sites like Yelp and TripAdvisor for example, they are already allow social search results in a hypertext search engine. Therefore social search engines will need a wholly different approach to searching to be effective.

Model Specific Formatted Search Results Using Thinking Sphinx

eric — Mon, 08 Feb 2010 14:15:23 +0000

Having recently implemented Thinking Sphinx on one of my web sites, I thought it would be cool to be able to search every indexed model. With Thinking Sphinx, it’s easy to have a bunch of different classes returned in the results. The tougher part is displaying them in a way that is organized (although admittedly not very DRY).

Let’s start out with the controller method for search results (app/controllers/search_controller.rb):

def result
if !params[:model].blank?
@model = params[:model]
else
@model = "ThinkingSphinx"
end
@query = params[:search][:query]

@results = Search.model_search(@model, @query)
end

Also, let’s make sure we define the actual search model. I define it in a separate search library (lib/search.rb). This is the relevant snippet of code:

def self.model_search(model, keywords, var = {})
@search_options = { :page => var[:page] || 1,
:per_page => 15 }

@search_options.merge!( :order => "@relevance DESC",
:sort_mode => :expr,
:sort_by => "@weight * @weight")

model.constantize.search(keywords, @search_options)
end

Assuming that we are working with the all models search, where the above model is ThinkingSphinx, let’s iterate over the search results with this code in the view (app/views/search/results.html.erb):

< % if @results.total_entries > 0 %>
<div id="localLocationList">
<ul class="list">
< % @results.each do |result| %>
<li>
< %= display_search_result(result) %>
li>
< % end %>
ul>
<div class="clear">div>
div>
< % end %>

And the view helper (app/helpers/search_helper.rb):

def display_search_result(result)
eval "render :partial => '#{result.class.to_s.downcase.pluralize}/result',
:locals => { :result => result }"
end

The interesting thing about this bit of code is the eval. The eval on each iteration decides which search result partial to display based on the class and then passes the result to the partial for display. So if the result has a class of Business, the partial app/views/businesses/_result.html.erb will be rendered. This is a quick example of a search result partial:

This is useful because all models don’t have the same characteristics. By creating a search result partial for each model type, this can be reused for consistent looking search results around your webapp. If the search is model specific, the same result partial will be used in every iteration over the results. If the search is model agnostic, then you can display your search results in a consistent manner.

SEO and Cross-Domain Content Syndication

eric — Wed, 03 Feb 2010 11:45:02 +0000

When dealing with content syndication, one is occasionally in the situation where you are not the higher ranking site in search engines. You might rank #4 for an article and in that same search, your syndicated content may be ranked #1. What’s the best way to deal with this?

After much reading and discussion, I believe that you have a few options.

The first is taking advantage of the most recent addition to the cross-domain rel=canonical link element. You can read up on it more here. If you can talk to the publishing group and get them to allow the addition of the canonical link relationship, then you will get credit where credit is due (and deserved).

The reason that this can sometimes be an issue is that a lot of SEO types are not comfortable with when and how to use this link relationship and over use it or point it to their homepage for more link juice. That’s not the idea here. The idea is to prevent duplicate content from being index and to allot credit to the deserving party.

The second is to try to take advantage of the syndicate publishing networks credibility and publish a blurb at the bottom of every article. Look at the bottom of every CNET article as an example. There is a blurb about every author with a link. That link could be to the author’s site or to an author’s bio page.

Lastly, you need to make sure that the article links back to your site. This is again taking advantage of the link juice and credibility that the higher ranking site can bring to your natural search ranking.

Being Smart is all about Being Resourceful

eric — Mon, 25 Jan 2010 12:45:48 +0000

The internet us the ability to not while keeping up the appearance that we do. Now that’s not to say that you should be a know it all, but you should definitely know how and where to get information if you need it. If you use a specific open source technology at work, then you need to know how to support (because odds are, it was written by a few interested people and doesn’t have a company behind it). So you should know where the forums are, where the documentation is, where the mailing lists and the mailing list archives are, etc. Do they have an IRC channel where you can talk to live users who might be able to help on a more immediate basis? Maybe there was an even a book written that you can get your hands on, a PDF, or even a screencast. If you lucky, you might write a Tweet about your frustration and one of the products creators will answer (which happened to me recently).

The best part about all this is the internet is the enabler. It’s searchable, indexed and provides you many methods of finding the answer to your (or other peoples) questions. Why am I telling you all this obvious information? Because its why I write my blog. People who need help need to be able to search for it. Most of the things I write about here are specific problems with answers that I figured out or aggregated from reading various things around the web. So be resourceful, use what’s out there and then document it. You may save yourself (and maybe me) a lot of time.

No related posts.