Back to blog

Designing our stack for high throughput and low latency.

2014-02-17 - Posted in Uncategorized Posted by:

by Rinat Abdullin 
For the last week I spent most of the time pairing with Pieter, learning more about our the performance and behaviour of our anticipated stack (for the second version of HappyPancake.com). It was thoroughly interesting exercise in systems engineering. Here is what our anticipated looks like right now: 2014 02 17 hpc design With this design we want to have 25ms latency of HTTP reads (non-cached, 99% percentile) given the throughput of 50000 requests per second. A/B testing, feature toggling, continuous delivery and live upgrades (with ghost mode and ramp-up) included. Here is a short summary of lessons learned within the last week:
  • Tomas is an absolute beast when it comes to crunching out small open source libraries
  • It is quite easy to publish statistics from an app and then gather them in a nice WebUI for crunching (using client library fsd to publish to local statsD app via UDP. StatsD currently pushes stats to Librato Metrics with delay of 10 seconds).
  • HTTP servers in Go are quite good, but can be a pain to extend or augment
  • Nanomsg is really nice and performant, however the documentation is … lacking.
  • Profiling capabilities of Golang are absolutely stunning.
  • Spending a week developing and debugging golang apps, while benchmarking them on a Digital Ocean cluster – teaches you a thing or two about Linux environment. It is awesome.
  • Software engineering is about making theories about how your code will behave in production, then running experiments to validate these theories. You iterate and learn.
  • Pairing up with somebody is an amazing opportunity to transfer knowledge and produce better quality code (I lost track of the number of times I was stunned and humbled by the experience and insight of other team members – so much to learn). We currently use TeamViewer (best image and keyboard sharing experience) and Skype for the voice. Campfire is for group chats (and chat ops).
For the upcoming week I’ll be working on pushing our stack closer to the desired performance numbers (we don’t meat the goal, yet). It is an interesting exercise which forces you to learn a lot and go deep (to the point of tuning the OS).

2 Comments

Joachim Rosskopf 5 years ago

Hey Rinat,

interesting read and great interview on the distributed podcast!

But some questions remain unanswered for me: I would be interested how the http/nanomsg transition in the terminator nodes and the matching of connection state from req. to resp. really works.

Perhaps you can elaborate on that a little bit more.
Thank you.

Joachim

Reply

Rinat Abdullin 5 years ago

Joachim, we plan to talk about that in more detail later.

In short:

1. Terminator gets a request, which is handled in its own goroutine (thanks to http package of golang). In this routine we serialize http request to message and push it to the bus. This go routine will wait till there is a response on the bus (or a timeout).
2. nanobus (our simple library on top of go-nanomsg) adds correlation ID to the message and pushes the message via BUS socket to all interested services.
3. Interested service receives a message, gets an http request, handles it (frequently using some other service down the chain to do something else) and then replies back via nanobus. Nanobus will use the same correlationID while pushing message to BUS socket.
4. Nanobus on the receiving side (terminator) matches incoming message to the outstanding request by correlationID and pushes message to associated go channel.
5. Waiting goroutine in http handler of terminator gets the message from the channel and writes it back to http connection, closing it. Alternatively it can timeout, writing “Internal server error” instead.

Reply

Leave a Reply

Your email address will not be published. Required fields are marked *