Another January, another Free Day come and gone. Win or lose, we hope everyone had fun, we sure had a lot of fun in the building here putting it on!
Let's go over some numbers, then I'm going to write a bit about how we managed to pull it off this year.
If you have experience with the first two Free Days, you might have noticed that this one went quite a bit smoother than in years past. Aside from an initial hiccup while traffic ramped up a full order of magnitude over a span of minutes, the site was responsive and snappy for the next hours as we dealt with the insane amounts of traffic. So what changed from years past?
We put to use some new software and hardware in our web stack. We now have two beefy web servers each with 16 cores and 32 GB of RAM, and a database server with double the RAM. In front of the web servers, we have redundant hardware load balancers to divvy up traffic to the two machines.
In addition to optimizing our own code base to run more efficiently on the hardware we've got, we changed some of the open-source software we use to run things. We jumped ship from Apache in favor of Nginx to serve web pages, with PHP-FPM doing the heavy lifting behind it. In front of everything, we are now running Varnish to cache pages where we can - which proved invaluable to putting on a successful Free Day with the traffic we saw.
When tracking down bugs from last year, we noticed that MySQL was a culprit more often than not when it came to slow-downs. In some places, we've replaced the use of MySQL with the document store MongoDB. For some data, including some hierarchichal data, it just made sense. This is one reason we can keep the page generation time for pages like yesterdays post, with over a thousand comments, so quick. Unfortunately, due to a small bug in Mongo's PHP driver, we experienced an initial slowdown until we could isolate the problem.
In the past, we served all assets from our own servers. During normal load, this was fine - pages were snappy and our bandwidth wasn't too outrageous. Of course, during Free Day everything changes :) Last year we were pushing much more bandwidth because we were serving all static content ourselves. Since then, we've changed things around to use Amazon's Cloudfront. Cloudfront is a global CDN that sits in front of the ever-popular S3 storage service that can deliver content to users from edge servers all over the world - and keep the bandwidth on their infrastructure instead of our own. This does put the bandwidth number above in perspective, though. The 76Mb/s we were pushing through our data center was almost entirely just gzipped text! That's a lot of bits!
Any computer scientist will tell you that random numbers that computers come up with are pseudorandom, that is they are still bound by their nature to be predictable. Even if there's a 1 in a million chance (or more), it's a chance nonetheless and can therefore be predicted beforehand. What, then, can we do to generate a completely random bidding system?
We decided on a Geiger tube to measure background radiation. The Geiger tube pulls a pin low every time it detects a piece of radiation, which for all intents and purposes is completely random and entirely unpredictable. We set up an Ethernet Pro to watch for these interrupts, and send a byte to a daemon running on one of our web servers to alert that the next participant was a winner.
Here's the Arduino code:
And for completeness, here's the bit server that Ben wrote in C to handle the extreme load of handling the hundreds of requests per second we saw on Free Day. He benchmarked it to something like 600,000 requests per second.
Love it or hate it, Free Day has come and gone another year. We in IT think it was success in that we were able to give away the money, and even under our own terms. We were aiming to stretch it out a bit more this year, rather than have it a flash in the pan and over with in just a few minutes. To that end, it was successful, and completely random.
I hope this gives some insight as to what goes on in the IT room at SparkFun during (our) busiest day of the year.
To all that won, congratulations! To those that didn't, thanks for trying and we all hope you had fun.