(Resolved) Networking outages caused by large DDoS attack

Unfortunately, it seems that the DDoS attacks have resumed.  Please see this updated post for more information.

Edit: This issue has been resolved; the attack seems to have subsided and service is back to normal.  The original post follows the break.

From the information available to me, including traceroutes and MTR reports taken at the time of the attack, this seems to have been a large-scale attack directed not at any particular person or server(s), but at Linode or perhaps even the datacenters themselves.  The attackers seem to have deliberately targeted critical network infrastructure in each datacenter, perhaps to the effect of causing as much disruption as possible.

Regardless, the attacks seem to have subsided, and service should now be back to normal.  Thanks everyone for your patience.


Original service announcement:

LizardNet is currently experiencing occasional networking outages due to what appears to be a very large-scale distributed denial-of-service (DDoS) attack that is targeting all of Linode, LizardNet’s provider.  The attack is extremely broad and has affected numerous different services across almost all of Linode’s various datacenters, and possibly even other providers at those datacenters.  Until the attacks can be mitigated, which can take quite some time and can’t really be predicted, networking outages may occur, and these outages may last for hours at a time.  Up to date information can be found on Linode’s status blog – remember that LizardNet’s servers are all in the Fremont datacenter.

Unfortunately, due to the nature of the attacks and the setup, this is entirely out of my hands – its up to the Linode and datacenter staff teams to mitigate the attacks, and I wish them luck.  Thank you for your patience and understanding, and for bearing with me and them.

If you are having trouble connecting to LizardNet services, simply try again later, perhaps in an hour or so.

(Note: LizardIRC has servers across various providers, though many of them are hosted by Linode in various datacenters around the world.  As the attack shifts from datacenter to datacenter, different LizardIRC servers will be affected.  For LizardIRC specific information, please refer to LizardIRC’s social networking pages: TwitterFacebookGoogle+.)

(Resolved) Networking outages caused by large DDoS attack

(Resolved) StartSSL OCSP Server Issues

Update: StartSSL’s OCSP responders appear to be up and running again, and are returning correct results.  This issue has been resolved.  If you disabled OCSP in your browser/software, it is safe (and recommended) for you to re-enable it now.

StartSSL, the company/Certification Authority I use to provide the security certificates for LizardNet (and LizardNet-hosted) sites and services, is currently performing some kind of maintenance or service upgrades.  Unfortunately, for whatever reason, this has interrupted their OCSP services, and as such any browser or program that does OCSP validation will report an error when attempting to establish a secure connection.  Firefox is known to be having issues because of this, presenting an error that looks like this:

An error occurred during a connection to domain.name. The OCSP server has no status for the certificate. (Error code: sec_error_ocsp_unknown_cert)

If your web browser gives you this message or a similar one, you can safely ignore the message and instruct your browser to continue to the site anyway.  The process to do this varies based on the browser, and I describe the process for the three most common browsers (Google Chrome, Mozilla Firefox, and Internet Explorer) at the end of this post.  If you’re having trouble establishing a secure connection in software that does not present security errors, for example the Thunderbird email client, you will need to (temporarily) disable OCSP checking entirely (see here for instructions to disable OCSP checking in Thunderbird).

As of this writing, StartSSL estimates that their services will be available again on Tuesday 22 December 2015 at 06:00 UTC.  Only time will tell if they’ll actually have things working again by then.


Bypassing certificate errors:

Google Chrome:

  • On the certificate error page, click the small “Advanced” link to the left of the “Back To Safety” button
  • Click the “Proceed to domain.name (unsafe)” link that appears

Mozilla Firefox:

  • On the certificate error page, click the “I Understand the Risks” link
  • Click the “Add Exception…” button
  • Uncheck “Permanently store this exception” if it is checked
  • Click the “Confirm Security Exception” button to proceed

Internet Explorer:

  • On the certificate error page, simply click the “Continue to this website (not recommended)” link.
(Resolved) StartSSL OCSP Server Issues

14 December 2015: Scheduled reboot for critical Xen security fixes

This is a past/expired downtime notification. The downtimes specified below have been completed, and remarks/results are given below as well.

Unless otherwise noted, all dates and times are given in Coordinated Universal Time (UTC), with time in 24-hour notation.

Once again, the Xen development team has released several highly-critical and so far undisclosed Xen Security Advisories (XSAs), and as such, Linode (LizardNet’s provider) will be performing emergency maintenance on all of their Xen hosts.  LizardNet’s sole remaining Xen system, phazon.fastlizard4.org, will be rebooted as part of the endeavour to patch the Xen vulnerabilities before the public disclosure date of 17 December 2015.  (More information can be found on the Linode status blog here.)

Update: The Xen Security team has lifted the embargo and publicly disclosed the vulnerabilities: XSA-155, XSA-157, XSA-164, XSA-165, and XSA-166.  Of these, XSA-155 and XSA-165 appear to be by far the most serious issues.  The QubesOS folks have published a commentary addressing these bugs.

The following server and services will experience downtime:

Date and time of downtime start: 10:00 Monday 14 December 2015 UTC (convert to other timezones)
Duration of downtime: Expected between 30 minutes and 1 hour, but up to 2 hours is possible
Status: Completed with no issues, despite late start (10:23 UTC)
Partial list of services affected:

  • LizardWiki
  • LizardNet OTRS (emails sent to OTRS during the downtime will be delivered after the downtime concludes)
  • LizardNet Continuous Integration (Jenkins) (Gerrit will not be able to trigger any jobs during the downtime, and they will not be run after the downtime concludes)
  • LizardNet Minecraft dynamic web maps
  • LizardIRC server emerald.lizardirc.org
  • LizardIRC’s website
  • LizardMail services on phazon.fastlizard4.org (emails sent to phazon.fastlizard4.org users during the downtime will be delivered after the downtime concludes)

Apologies for the short notice on this downtime (both from me and Linode).

14 December 2015: Scheduled reboot for critical Xen security fixes