So, it looks like they're using BGP to "program" the adjacent router a la "SDN." I'm curious as to two things:

* Byzantine fault-tolerance: How does this system handle failures when a specific node fails in a way that it fails to withdraw its routes. When a node's haproxy fails, how is BIRD informed of its failure? What if the failure is in some way that internal fault detectors don't see the failure.

* How is the ECMP hashing problem handled? ECMP hashing on most gear is just a plain hash, that means when a route is withdrawn, the rest of the systems see their traffic rebalance. How does this not result in all connections being severed?

Hi Michael, you wrote a whole post about varnish cache management without mentioning hit-rates! How effective was all this? How many of that 23K req/s did origin have to handle?

Random note on on scale-out, caching, etc: check out CloudFront's "whole site delivery". You can set min TTLs to zero on CloudFront, configure your cache headers correctly and get many of the benefits outlined in the article. See:

   http://www.allthingsdistributed.com/2012/05/cloudfront-dynamic-content-support.html

   http://blogs.gartner.com/lydia_leong/2012/05/14/amazon-cloudfront-gets-whole-site-delivery-and-acceleration/

WARNING: CloudFront will dutifully cache non-2xx responses, so you can get a long-lived, but very fast, 500 response...

If I was to use Varnish, would that make my need for Redis obsolete - should I switch to a non-memory based key-value db and use Varnish on top of that?

My current stack: nginx/php-fpm/redis. Nginx and Redis serve me well, but php-fpm makes my website rather slow with high volumes of traffic, so I believe the solution for that would be Varnish(?).

>278 employees. That's a lot of people, I imagined they were way less.

Apologies if this is not of interest.

If you found this interesting please checkout the jobs page [1] at Tumblr, we are constantly looking for new folks. Specifically [2] for positions on the teams that implemented everything described in the article.

[1] https://www.tumblr.com/jobs [2] http://boards.greenhouse.io/tumblr/jobs/17886