Beginning on January 9, 2015 Pressable experienced the worst outage in the history of the company. Below is an email from Vid Luther (Founder and CEO) sent to all customers impacted by the issue.
On behalf of the entire team at Pressable, I would like to say we’re sorry.
We’re sorry about the outages you have experienced recently, including the longest ever outage in company history, this weekend.
I’m also sorry for not responding to some of your phone calls and personal emails inquiring about the issues. While my team worked dilligently throughout the weekend, updating customers via social media, email, and our helpdesk, there was a lack of transparency by my radio silence.
That wasn’t our intent, but being such a small team, we had to prioritize on restoring functionality.
Now that it’s been over 24 hours of normal service, I wanted to share what happened, and what we’re doing about it.
The simple answer is that we could not keep up with our growth, our primary caching system reached it’s limit, and when it failed, the sheer number of requests going through our system flooded our internal systems. Currently our system has an internal limit of 1 Gbps, and we saw sustained traffic 5 times that. In a way, we DDOS’d ourselves.
This is also why we erroneously reported that we were under a bot attack at one point.
What are we doing about this?
Fortunately, this is something that wasn’t completely unanticipated, we had identified this as a potential issue last summer, and had been working on upgrading our systems over the next two months.
We will double our efforts to finish this migration sooner than later.
On behalf of the entire team at Pressable, I apologize for the incident this weekend, and would like to assure you that we are working around the clock to prevent this from happening again.
Founder and CEO