Denial of service attack December 26th - Network Migration (Completed)

JasonJason CEOCheckfront
edited January 2016 in News and Updates
At 8am on December 26th 2015 our upstream hosting provider was under a distributed denial of service attack (DDoS). This started in our us-central network based in Dallas TX. This was mitigated within the hour but resulted in partial down time and degraded performance.

Around 10pm in the evening the attack resumed, focused in succession on the remaining data centres starting with our UK network, moving onto us-west (Fremont CA), us-east (Newark NJ). Our Asia network was not targeted. As each attack was mitigated the next location would begin. This attacked continued till approximately 7am PST the next morning. We continued to update our network status page at http://status.checkfront.net/

 During that time performance was impacted, and in some cases service was unavailable intermittently depending on the location and focus of the attack. DDoS attacks are unfortunately very difficult to mitigate quickly. They are malicious by design and are initiated from thousands of computers worldwide. As part of our redundancy plan we are able to re-route traffic from one network to another geographic location, however due to the round-robin design of this network attack that was not a viable option.

We are currently reviewing our redundancy plan and infrastructure to ensure such attacks have minimal disruption to service as possible. We don’t have any additional information at this time as to the source of the attack or the motive. With that said we are confidant this attack was not directly aimed at Checkfront or our customers, but the upstream network as a whole.

We continue to monitor the situation and work with our upstream provider. Our apologies to any customers impacted and I’ll update this tread when I have more information.

 -Jason

Comments

  • JasonJason CEO Checkfront
    edited December 2015
    To update this attack has resumed mainly on our us-central network (6am PST Dec 29).  We are currently in the process of transitioning to a new hosting provider (google cloud) to stop any further impact to our customers.  I'll update here when this is complete.  
  • JasonJason CEO Checkfront
    We have moved our largest group of customers off our existing hosting provider to a new network outside of our current provider.  This should prevent further impact from the larger upstream denial of service attack happening this week. Other customer groups will follow, however this one in particular was the most impacted.  We'll continue to monitor.  

    Thanks for your patience and we apologize for any service interruption that may have occurred.
  • JasonJason CEO Checkfront
    edited January 2016
    We are beginning the process of migrating the rest of our customers to our new Google Cloud network.  This includes our remaining us-west, us-east, uk and asia networks.  We will do this in the order we feel is the most vulnerable to the current DDoS attacks with our current provider.

    Migrated: us-central1 - Dec 28
    Migrated: uk1 - Dec 31
    Migrated: uk3 - Dec 31
    Migrated: uk2 - Dec 31
    Migrated: us-central2 - Dec 31
    Migrated: us-central1 - Dec 31
    Migrated: asia1 - Jan 1st
    Migrated: us-east1  - Jan 1
    Migrated: us-east2  - Jan 1
    Migrated: us-east3 - Jan 1
    Migrated: us-west1 - Jan 1
    Migrated: us-west2- Jan 1
    Migrated: www.checkfront.com - Jan 1
    Migrated
    : single instance client nodes  - Jan 1

    All customers migrated as if Jan 1 2016.
  • JasonJason CEO Checkfront
    edited January 2016
    For those interested in a technical explanation of the DDoS attack over the last week at our current network provider, please see the below excerpt from them.  To be clear we are still continuing to migrate the rest of our platform to Google Cloud.  Our remaining customers will be transitioned over the next few days or sooner if needed.  Linode has been great to us, but we know how critical performance and uptime is to your business, so these steps need to be taken despite best efforts all around.

    No amount of downtime or performance degradation is acceptable to us.  All of our staff have been working 24/7 since this started on Christmas day to ensure your questions are answered quickly.  During that time our technical team has been busy provisioning the new network with minimal or no downtime to the majority of our customers.  Migrating thousands of businesses spread across 14 regional servers to a brand new network in a few days is complicated to say the least. Thanks again for your patience, and a personal thanks to my staff for their extra efforts.

    No network, be it Amazon, Google, Rackspace or others large providers are invulnerable to malicious attacks.  But the DDoS mitigation on Google Cloud is the same technology used to defend their own systems.  We believe this is a great step forward in ensuring the performance and availability of your Checkfront account.  

    I'll update above when the remaining customers on our network have been transitioned over.  As of this point over 70% of our customers have been migrated, and most of those remaining have yet to experience any service degradation from the DDoS attack.  Service alerts will continue to be posted on http://status.checkfront.net

    Best,
    -Jason
    Co-founder, CEO - Checkfront

    "I’d like to share some updates about the recent DDoS attacks. 

    I am one of several network engineers at Linode who have been working around the clock on DDoS mitigation. While things are stable, I would like to take a moment to publicly address the large and frequent DDoS attacks that we have been receiving since Christmas Day. 

    It has become evident in the past two days that a bad actor is purchasing large amounts of botnet capacity in an attempt to significantly damage Linode’s business. The following is a partial list of attacks we have received in no particular order: 

    - Multiple volumetric attacks simultaneously directed toward all of our authoritative nameservers, causing DNS hosting outages (not used by Checkfront)

    - Multiple volumetric attacks simultaneously directed toward all of our public-facing websites, causing Linode Manager outages (no impact to Checkfront accounts)

    - Layer 7 (“400 bad request”) attacks toward our web and application servers, causing Linode Manager outages  (no impact to Checkfront accounts)

    - Large volumetric attacks toward our colocation provider’s upstream interconnection points, overwhelming the router control planes and causing significant congestion/packet loss 

    - Large volumetric attacks toward Linode network infrastructure, overwhelming the router control planes and causing significant congestion/packet loss 

    All of these attacks have occurred multiple times. Over the course of the last week, we have seen over 30 attacks of significant duration and impact. As we have found ways to mitigate these attacks, the vectors used inevitably change. 

    As of this afternoon, we have mostly hardened ourselves against the above attack vectors, but we expect more to come. We are working extremely closely with all of our technical partners, including our network equipment vendors and our colocation providers, to prevent future attacks. 

    Once these attacks stop, we plan to share a complete technical explanation about what has been happening. Additionally, we will be announcing the details of an ongoing project to significantly improve our internet connectivity and resiliency. "

    As always stay tuned here and on http://status.checkfront.net.  

    -J
  • too early to say job well done?
  • JasonJason CEO Checkfront
    So far so good - thanks.  All customers were migrated as of yesterday (Jan 1). 
This discussion has been closed.