1. Collection types
2. Virtual Nodes
3. Row caching

The cool thing about collection type support is that it is allowing Cassandra to become more versatile for storing different types of data (read more about collections on the Cassandra developers blog here). With the addition of sets, lists and maps, many organizations will be able to move some of their caching and data organization off of Redis and into Cassandra.

Along the idea of caching is the new row-based caches. This works out incredibly well as it removes the need for (some) machines acting as a caching layer in front of Cassandra. It is not an uncommon paradigm to cache the output of an entire row of data in a caching layer. Having C* do this natively would allow the caching layer to be reserved for more complex objects. Additionally, having row caching inside Cassandra means that when a node coordinating a query hits the other nodes where it would expect to find the result, the node with the result already in its query will respond first and quickly. Meanwhile, the other node(s) will not need to respond to the coordinator and still have the result added to the row cache for future use.

Last but not least is the idea of virtual nodes. For a lot more information than this, I recommend watching Sam Overton’s talk on them from the summit. In short, virtual nodes make token management basically unneeded, improves bootstrapping and decommission speed, and make incremental cluster growing and shrinking possible. And considering how often anyone with a cluster of > 15 nodes does all of those things, this is a next level feature to be on the lookout for.

For more information, check out the blog post I wrote (which talks a little less about the roadmap) on the SimpleReach blog entitled A Big Stage for Cassandra.

And if you don’t want to click through, here is are the 2 videos that I am a part of. The first video is my talk at the summit. Topic of my talk is Polyglotteny. This is the notion that is acceptable to use more than one language and more than one data store to support your applications various needs. It is a little tour through how SimpleReach started out with Ruby and Mongo and ended up at Ruby/Node.js/Python and Mongo/Redis/Infobright/Cassandra.

For fun, here are the slides that go along with the talk (some of which you can see on the video).

This second video is of Russell Bradberry (Principal Architect at SimpleReach) and I talking to John Furrier of Silicon Angle. We talk a little about SimpleReach and how we use Cassandra to manage our data.

All in all the summit was a great experience, both in getting to share ourselves (SimpleReach) with the community and to meet and greet with the people that make it happen. Datastax was also kind enough to name me one of their MVPs for being a participating part of the Cassandra community. Cassandra has really made a name for itself and it’s great to be a part of the community behind that name.

You can do something like:

But making storage more complex isn’t a great thing either. So how does adding more to a name make things easier? Think of it as a method of grouping. So let’s say you want to store information about a blog post URL (and for the sake of example, let’s say that the hashed UUID version of the URL is the rowkey). How do you break up what’s important about the URL into storage? In a common WordPress post, you have tags, categories, published date, title, URL, id, content, author(s), and even custom fields. So I would structure that as follows:

Now that we have a layout, let’s see why it’s good to have it broken up like this. If you want to load up an edit screen or render the page, then you can grab the entire row. But if you just want part of the row (which is a great thing when you have much wider rows), then you can do things like just ask Cassandra for the meta: data or just the tag:‘s data. Like any other data store, Cassandra returns the data more efficiently when being asked for something smaller and more specific than just an entire row.

I know this is a bit of a contrived example and therefore is overly simplistic. But the idea here is to show how one can extrapolate a storage methodology for a column family using composite columns. So in a bit more complex of an example, think of an event tracking system where we are going to be tracking things based on a URL (also in hashed UUID form as above). So we’ll build a composite row key of the timestamp in milliseconds since epoch of the beginning of the day and the hashed URL UUID. Note: I’m going to ignore the fact for examples sake that this row key methodology could create hotspots in the cluster.

What this allows you to do is to create queries where you say, “give me all pageviews that took place between 2am and 3am (in milliseconds since epoch form). I’ve also only put pageviews and a single custom event in the example above. But you can imagine a situation where you can create any number of event types and have the event type be the first comparator in the CompositeType(), the timestamp can be the second comparator, and the third is a Time UUID so that if you have more than one column per event, you can grab all the columns for a particular event.

As you can see by the two over-simplified examples above, composite columns in Cassandra have the potential to be very powerful in you know how your data is going to look and how you’d like to access it. As you note in the second example, you can’t find all events that happened between 2am and 3am with a single query. The only option is to create a loop with the list of event types and slice out the events per time period for each event type (in each iteration of the loop). So knowing your data access patterns is important for creating your data storage patterns. There may be a little trial and error involved as you scale and add more data. But knowing the tools that are available to you is an incredibly important step in that process.

For the sake of terminology, a job in Resque is a message in SQS. A big issue with using SQS is that if you need more than 120k allowed messages to be in-flight at a time, you have to request an increase. As of the publishing of this post, it has to be a special request made through an account manager or through support. This is easy and nothing other than a potential inconvenience worth noting. Another feature that has the potential to put a kink in your use/case is the fact that you can only pull off 10 messages at a time. In order words, 10 is the max batch size per receive request. So if you regularly put 1 million messages on a queue, then pulling off 10 messages off at a time would take 100k jobs. A potential deal breaker is that you can’t yet flush a queue in SQS. So if your queues are a little backed up?? and you want to start over, the only choice you have is to delete your queue (which will remove any custom settings that the SQS team may have made).

Another thing that most people tend to overlook is the cost of running each system. Let’s assume that we are running everything in AWS US East. The actual machines running Redis/Resque don’t need a lot of horsepower or a lot of memory. So for arguments sake, you can run the master/slave combo for Redis on two medium instances which amounts to roughly $250 per month. The thing you need to remember about Redis is that is can only use one core. Note here in the documentation (http://redis.io/topics/benchmarks):

Redis is a single-threaded server. It is not designed to benefit from multiple CPU cores. People are supposed to launch several Redis instances to scale out on several cores if needed. It is not really fair to compare one single Redis instance to a multi-threaded data store.

So if you plan on using Redis for something else in addition to Resque (i.e. as a caching system), I recommend another EC2 configuration with Redis instance counts equal to the number of cores on the machine (minus one for of the OS). Right now SQS charges $0.120 per GB data transfer out and $0.01 per 10,000 Amazon SQS Requests ($0.000001 per Request). Assuming we handling 10 million messages per day (which is 300 million messages per month) at roughly 1k per message (or 300G transfer per month). In SQS this would cost about $350.

So right off the bat on a price comparison, Redis wins if you are only looking at cost. However, that also means building and administering 2 Redis servers which also comes at a cost. But you may be paying even more than the $350 dollars per month if only pulling 10 messages at a time per batch requires you spin up more workers. Something else that may be an additional cost is that using SQS may require you to use SNS (Amazon’s Simple Notification Service). If you aren’t using SNS and don’t know how, then monitoring your queues might have an additional learning curve associated with it (as opposed to one of many Nagios scripts that will check Resque queue sizes).

As with anything else, your mileage may vary. So learn your use/case and what compromises you may need to make when going with either one system or the other.

Note: This post is intended to be a comparison based on recent experiences. Amazon’s SQS team has been fantastic throughout the (investigation) process in discussing their roadmap (when possible) and giving the account special dispensation where possible.

But lately we’ve expanded those talks to include topics like, “How Does Venture Capital Work?” and “What is the History of SimpleReach?” By including the business side of the organization, we build a better more cohesive team. We also ensure that the transparency that we incorporate into the culture isn’t just transparency for the sake of being able to say so, but also that the transparency is understood by everyone who desires to understand. Very few engineers actually understand some of the business side of things (like venture capital). And by ensuring that everyone in the company knows the history and evolution of the company, they can better understand and contribute to the vision and shaping of the organization.

These are just a few of the topics that we cover in our sessions. We don’t relegate the topics to engineering or business, those just end up being the main topics because those are in the house expertises and interests that everyone shares. Knowing who you can go to for additional help on something you are working on or finding new passions within an organization are great motivators. It’s also help us find some issues with some our current architecture. So you take the good with the bad because it helps drive company forward. But either way, the best part is that everyone walks out of the room more educated than they entered. And the sharing of information is a great unifier of teams and one of the reasons that Continuous Learning is such an important part of a good startup culture.

The more interesting thing to note is not that we ended up with Cassandra, but what we used to make that decision. When it came down to Cassandra and HBase, they both had their pros and cons for our use/case. In fact, it’s likely that either one of them would have worked out just fine in the long run. We actually made our decision based on the community and the roadmap (but mostly the roadmap).

The product roadmap is simply something that says where the product is intending to be in the next few weeks, months or years. And it can be important because if the product roadmap in 12 months is in alignment with where you see your technology in 12 months, then it would seem like a pretty good fit. And it potentially allows your organization to help influence the development of new technologies. I’m sure some of you are thinking that it’s no fun being the guinea pig or being on the bleeding edge of everything. But when you are pushing the limits of today’s technology stacks and applications, you have to be on the cutting edge every now and then.

Product roadmap doesn’t just tell you about the application itself, it tells you about the community. If you see yourself aligned with the roadmap in the next 12-24 months, then you are also aligning yourself with the community. And the community is hopefully full of people that have a like-minded set of goals for the use of the product in question. So don’t just think about what’s good for you now, think about what going to be good for you (or your organization) in the future too.

Looking at this from a slightly different perspective, Google is saying that if you just make your site SSL available, then you can continue to have your referrers. And that is ultimately what people (read marketers and SEO folks) want anyway. To oversimplify a bit, making one’s site available over SSL is as easy as going to GoDaddy or the like and buying and installing an SSL certificate on your web server.

But what does having this certificate really do? It allows a website to be loaded in a secure, encrypted environment. It also allows the browser and the user to validate that the site is who they say they are according to a set of authorities like Verisign or Thawte. These are the folks whose job it is to verify that the certificate is being issued to a valid company (note that I said valid, not necessarily reputable as it’s not the job of certificate authorities to determine reputation).

And on a more technical level, as a user, SSL certificates keep traffic between you and the website you are interacting with more secure. Looking at this via the OSI model for networking; since all HTTP traffic happens at the application layer (layer 7), when SSL is not present, everything happens over plain text communications and can be sniffed. SSL, which is a network protocol, occurs at layer 6 (the presentation layer) and therefore can encrypt and decrypt all the communications that happen at layer 7 (if used).

So if we all bit the bullet and added SSL capabilities to our sites, the net result would be a more secure internet from a user perspective. There are plenty worse things that Google could be doing than forcibly making the internet more secure.

So I took my SysAdmin based love of making things quicker and learned enough Applescript to let Alfred make my life easier. I wrote a script that when a domain argument is passed to it in Alfred, it will open up a Quantcast tab, a Compete tab, and an Alexa tab for that domain in Google Chrome (Note: With the script below, it MUST be Google Chrome). To install, go to the Alfred preferences and create a new Applescript extension.

Fill out the text boxes as follows:
Title: domstat
Description: Get the domain statistics for
Keyword: domstat

Now put this in the script box:

To execute, just fire up Alfred and type: “domstat eric.lubow.org” and it will fire up Google Chrome and open up the tabs.

Update:
You can even download the Alfred extension directly from here.

The error looks something like this:

The reason for the error is because CentOS (at least the RightScale version 5.6.8.1 has an old certificate authority bundle: /etc/pki/tls/certs/ca-bundle.crt.

I backed up the existing certificate file just to be on the safe side.

To fix the issue, just download a new certificate bundle. I used the one from haxx.se.

In the usual spirit of social coding, I have added the script to Github: https://github.com/elubow/ec2-consistent-snapshot.

Running it is just this:

The options used here (for reference) are telling ec2-consistent-snapshot to use –mongo, on the –xfs-filesystem /data, in the us-east-1 –region (note that it’s just the region and not the availability zone within that region), to be backed up with the listed –description of the specified volumes. You can even throw a –mongo-stop in there to have Mongo stopped before the file system freeze and then restarted after the volumes have been backed up. Don’t forget that you need to set your Amazon keys in you environment variables (AMAZON_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY for your key and secret respectively).

I attempted to keep the usage style consistent with Eric Hammond’s original version, just add Mongo support for it.

Note: I also mentioned this on the mailing list. But given the amount of messages that fly around on the list daily, some folks may have missed it.

References:

• ec2-consistent-snapshot blog entry by Eric Hammond
• ec2-consistent-snapshot on Github with Mongo DB support
• Backing up MongoDB on EC2 (10gen)

1. Boxcar

   
   Just is case you haven’t heard of Boxcar, it’s what notifications for the iPhone should have been. You can get push notifications for a ton of different services ranging from Facebook, Twitter, and email to Github or even something more custom (for those of you techies who read this blog). This awesome iPhone application has recently been released for Mac desktop. This means that those same notifications that you used to have to have tabs open for Facebook, Twitter, RSS feeds, Email, Github, or whatever other services you use are now all centrally located. Boxcar for Mac is still beta-ish so expect it to get a lot better. But centralized notifications helps to prevent you from checking all 80,000 (or so) locations for new items to distract you.

2. Alfred

   
   Alfred App is what Spotlight should have been plus some. It is by far the application that I use the most on my Mac. It means that I grep through files, search my entire filesystem and either open a file or find the containing folder and open it up. And with the Powerpack you have the clipboard manager (which happens to be my favorite feature). It does favorite snippets and can save old clipboard contents for long period of times that can searchable. If you try Alfred and it doesn’t make your life easier, then you are using it wrong. I could go on for hours with how Alfred can make your Mac life better, but it’d faster and easier to just read the Tips Blog.

3. Caffeine

   
   Caffeine is not really a productivity app, but something more to prevent annoyance and generally an all around handy app to have. It does one thing and does it well. Caffeine prevents your computer from going to sleep. This is great if you have a short screen saver that you don’t feel like changing or if you are watching a movie on Netflix and don’t want your computer to go to sleep. There is something to be said for simplicity and doing something well.

4. Notational Velocity

   
   Mac sticky post-it style notes are good, but Notational Velocity has taken it to the next level. It’s freeform, searchable, remote-syncable, taggable notes (too many buzzwords, right?). But the fact is, you just start typing and it saves as you go. When you are done, you can add tags. And if you have an iPhone, then you can install SimpleNote and have your notes from the computer sync’d to your phone (and vise versa). But my favorite thing is just the fact that you can start typing and it is immediately searchable. I have it open on all my spaces and I am constantly making notes. I take items from my Alfred clipboard and paste them into NV as notes for how I get stuff working. This way I keep track of everything I tried and then just remove the things I don’t use (and then use those notes to write a blog post).

5. Homebrew

   HOMEBREW

   I have tried the gamut of package management for the Mac. I compiled things from source (and that just gets messy). I have also tried Fink and Macports and they just both felt a little hackish given the naturally usable feel of OS X in general. So I installed Homebrew and everything just sort of fell into place. It’s just as simple as “brew install $package” (after Homebrew is installed of course). And since every package installed is installed in isolation (/usr/local/Cellar), removing and upgrading can also be done with ease. If there was a solid GUI in front of it, I would recommend Apple adopt it as a 3rd party package management system.

If there are other packages or apps for the Mac that has had a great impact on your productivity, let me know.