(Resolved) Resumed/ongoing DDoS attacks targeting Linode infrastructure causing service interruptions

Update 14 January 2016: The attacks seem to have finally subsided and this issue is now resolved.  Linode has not reported any signs of the attack for a few days now, and they to have declared the incident to be over.  They’ll be publishing a full report on the attacks soon, and I’ll update this post when that becomes available.

All LizardNet services should now be operating normally, which no further risk of downtime or interruptions caused by the attacks.  Thanks for your patience!

Update 6 January 2016: The attacks against Linode are, unfortunately, still ongoing, though it seems that the network engineers have made good headway in mitigating and hardening against the attacks.  No significant service disruptions have occurred for over a week now; the most that has been seen is occasional slow performance due to increased latency or packet loss.  Besides, that, though, everything seems to be operating mostly smoothly.  Of course, until the attacks either cease or are completely mitigated against (which will still take some time yet), the chance remains still of occasional slow/degraded performance, along with a slight chance of temporary outages (though, based on the pattern, no further outages are expected as of this update).

In other words, expect perhaps some occasional slowness and nothing more, though don’t be too surprised if outages start occurring again if the attacks shift.

Original post: Unfortunately, the DDoS attacks targeting service provider Linode’s infrastructure have resumed and are ongoing.  According to a preliminary report released by Linode, since Christmas Day, Linode has received over 30 attacks “of significant duration and impact”.  Linode’s network engineers are working around the clock to mitigate the attacks, however, it is inevitable that the attacks will cause service interruptions ranging from degraded performance to full outages of LizardNet and LizardNet-hosted sites and services.  Hopefully, as attack vectors are mitigated, the interruptions will become less frequent and severe, but until the attacks cease, it’s worth noting that service interruptions may occur, though hopefully not as often or as severely now that network protective measures are in place.

Fortunately, it seems that the Fremont datacenter, which houses LizardNet’s servers, has been spared the brunt of the attacks, or for some reason has been better able to cope with them than some other datacenters.  This morning there was a period of an hour or two of increased latency and packet loss, but otherwise all LizardNet services were still available.  That doesn’t rule out future service interruptions, though, so if you start having trouble accessing LizardNet services, it’s almost certainly due to a shift in the ongoing attacks.

As before, this is out of my hands and there’s nothing that can be done except to wish Linode and the other upstream service providers luck in defending against these attacks.  It’s worth noting that this is an extremely massive attack, targeting networking infrastructure both at Linode’s datacenters and at upstream interconnection points; indeed, I would even hazard to call these attacks unprecedented in severity, coordination, persistence, and duration.

Linode has indicated that they plan to publish a detailed report once the attacks are fully mitigated and/or cease, which will allow for a more detailed analysis of the attacks.  Until then, though, thank you for bearing with me.

Best of luck to the Linode network engineering teams!

(Note: LizardIRC has servers outside of the Fremont datacenter and with other non-Linode providers; for more information specific to LizardIRC, please visit LizardIRC’s social networking pages: TwitterFacebookGoogle+.)

(Resolved) Resumed/ongoing DDoS attacks targeting Linode infrastructure causing service interruptions

(Resolved) Networking outages caused by large DDoS attack

Unfortunately, it seems that the DDoS attacks have resumed.  Please see this updated post for more information.

Edit: This issue has been resolved; the attack seems to have subsided and service is back to normal.  The original post follows the break.

From the information available to me, including traceroutes and MTR reports taken at the time of the attack, this seems to have been a large-scale attack directed not at any particular person or server(s), but at Linode or perhaps even the datacenters themselves.  The attackers seem to have deliberately targeted critical network infrastructure in each datacenter, perhaps to the effect of causing as much disruption as possible.

Regardless, the attacks seem to have subsided, and service should now be back to normal.  Thanks everyone for your patience.


Original service announcement:

LizardNet is currently experiencing occasional networking outages due to what appears to be a very large-scale distributed denial-of-service (DDoS) attack that is targeting all of Linode, LizardNet’s provider.  The attack is extremely broad and has affected numerous different services across almost all of Linode’s various datacenters, and possibly even other providers at those datacenters.  Until the attacks can be mitigated, which can take quite some time and can’t really be predicted, networking outages may occur, and these outages may last for hours at a time.  Up to date information can be found on Linode’s status blog – remember that LizardNet’s servers are all in the Fremont datacenter.

Unfortunately, due to the nature of the attacks and the setup, this is entirely out of my hands – its up to the Linode and datacenter staff teams to mitigate the attacks, and I wish them luck.  Thank you for your patience and understanding, and for bearing with me and them.

If you are having trouble connecting to LizardNet services, simply try again later, perhaps in an hour or so.

(Note: LizardIRC has servers across various providers, though many of them are hosted by Linode in various datacenters around the world.  As the attack shifts from datacenter to datacenter, different LizardIRC servers will be affected.  For LizardIRC specific information, please refer to LizardIRC’s social networking pages: TwitterFacebookGoogle+.)

(Resolved) Networking outages caused by large DDoS attack

(Resolved) StartSSL OCSP Server Issues

Update: StartSSL’s OCSP responders appear to be up and running again, and are returning correct results.  This issue has been resolved.  If you disabled OCSP in your browser/software, it is safe (and recommended) for you to re-enable it now.

StartSSL, the company/Certification Authority I use to provide the security certificates for LizardNet (and LizardNet-hosted) sites and services, is currently performing some kind of maintenance or service upgrades.  Unfortunately, for whatever reason, this has interrupted their OCSP services, and as such any browser or program that does OCSP validation will report an error when attempting to establish a secure connection.  Firefox is known to be having issues because of this, presenting an error that looks like this:

An error occurred during a connection to domain.name. The OCSP server has no status for the certificate. (Error code: sec_error_ocsp_unknown_cert)

If your web browser gives you this message or a similar one, you can safely ignore the message and instruct your browser to continue to the site anyway.  The process to do this varies based on the browser, and I describe the process for the three most common browsers (Google Chrome, Mozilla Firefox, and Internet Explorer) at the end of this post.  If you’re having trouble establishing a secure connection in software that does not present security errors, for example the Thunderbird email client, you will need to (temporarily) disable OCSP checking entirely (see here for instructions to disable OCSP checking in Thunderbird).

As of this writing, StartSSL estimates that their services will be available again on Tuesday 22 December 2015 at 06:00 UTC.  Only time will tell if they’ll actually have things working again by then.


Bypassing certificate errors:

Google Chrome:

  • On the certificate error page, click the small “Advanced” link to the left of the “Back To Safety” button
  • Click the “Proceed to domain.name (unsafe)” link that appears

Mozilla Firefox:

  • On the certificate error page, click the “I Understand the Risks” link
  • Click the “Add Exception…” button
  • Uncheck “Permanently store this exception” if it is checked
  • Click the “Confirm Security Exception” button to proceed

Internet Explorer:

  • On the certificate error page, simply click the “Continue to this website (not recommended)” link.
(Resolved) StartSSL OCSP Server Issues