Queue system at Wuaki.tv
As the number one Video on Demand Service in Spain Wuaki.tv has faced lots of interesting challenges, one of them is to make our application as asynchronous as possible. In order to avoid any issue that could cause that our application gets blocked during long periods of time, we needed to find a way to avoid any third party related issue.
As you can imagine from our previous posts at Wuaki.tv we are heavy users of Redis, and this was the very first project were we introduced Redis as our backend storage, not only to our stack but to part of our team.
The first step we made in order to become asynchronous, was to create a queue stack built on top of Resque and Redis, at that moment we thought that Redis was more than capable of handling the amount of requests that we were planning to send to it, and given its speed in conjuction with Resque our work was practicly done in a easy and fashionable way.
During 2012, wuaki.tv was growing exponentially, and our stack started to become more complex almost every two or three weeks (we still change very rapidly), and we began to develop new services as we started a race to convert our service into a micro-service oriented application. At the beginning we thought that the best choice was to create a queue stack for each micro-service, with independant Redis instances, but after a couple of months we did realise that Redis powerful enough to handle all this work and that our stack was becoming really complex so we decided to centralise some of our services and reduce complexity. After we found out our company business plans for the next 2 years, we came to the conclusion that maintaining this stack was going to be really painful and that the way we were implementing this wasn’t going to scale, so we went back to the whiteboard. After talking to Juan Hernández and Toni Reina for a couple of hours, we came to the conclusion that creating just one stack for async tasks was more than enough and if we combined this with Amazon autoscaling features we could build a really reliable stack.
Once we have agreed the work that had to be done, was time to design the infrastructure and software that was going to help us solve this problem. In order to accomplish all this, we built a Async task stack on top of Nginx + Unicorn + Resque + Resque web + Puppet and AWS Autoscaling, Toni Reina was kind enough to provide a custom config.ru inside of Resque-web, so we can load and connect to different Redis instances from only one application, depending on the URL. We developed a Resque web module for puppet which deploys Resque web on your servers while we scale up our infrastructure managing:
- Deploy Resque web
- Manage Config.ru template file
- Manage Unicorn configuration
At the moment, Wuaki.tv is built on top of 7 different applications and APIs, so we’ve decided that the easiest way to identify which job is going to be processed by which queue we should add a namespace to our Redis database. This is helping us reduce the pain of scaling and managing this stack.
If you think about this method as your solution to concurrency and resilience problems, please take into consideration that this carries a change of paradigm, when you develop queuing system you should must get rid of long and intensive CPU jobs because the idea of this is to scale and process tasks in a more efficient way, enough of the bulk jobs.
The idea behind our use of Async Tasks is to deliver the best user experience, taking away any task that could cause performance issues.