20 November 2016: Scheduled reboot for critical Xen security fix

This is a past/expired downtime notification. The downtimes specified below have been completed, and remarks/results are given below as well.

Unless otherwise noted, all dates and times are given in Coordinated Universal Time (UTC), with time in 24-hour notation.

The Xen development team has released several critical and so far undisclosed Xen Security Advisories (XSAs), and as such, Linode (LizardNet’s provider) will be performing emergency maintenance on all of their Xen hosts.  LizardNet’s sole Xen system, phazon.fastlizard4.org, will be rebooted as part of the endeavour to patch the Xen vulnerabilities before the public disclosure date of 22 November 2016.  (More information can be found on the Linode status blog here.)

The following server and services will experience downtime:

phazon.fastlizard4.org
Date and time of downtime start: 12:00 Sunday 20 November 2016 UTC (convert to other timezones)
Duration of downtime: Expected between 30 minutes and 1 hour, but up to 2 hours is possible
Status: Completed on schedule with no issues!
Partial list of services affected:

  • LizardWiki
  • LizardNet OTRS (emails sent to OTRS during the downtime will be delivered after the downtime concludes)
  • LizardNet Continuous Integration (Jenkins) (Gerrit will not be able to trigger any jobs during the downtime, and they will not be run after the downtime concludes)
  • LizardNet Minecraft dynamic web maps
  • LizardIRC server emerald.lizardirc.org
  • LizardIRC’s website
  • LizardMail services on phazon.fastlizard4.org (emails sent to phazon.fastlizard4.org users during the downtime will be delivered after the downtime concludes)

Apologies for the short notice on this downtime (both from me and Linode).

Advertisements
20 November 2016: Scheduled reboot for critical Xen security fix

25 October 2016: Emergency reboots to patch “Dirty Cow” vulnerability

This is a past/expired downtime notification. The downtimes specified below have been completed, and remarks/results are given below as well.

Unless otherwise noted, all dates and times are given in Coordinated Universal Time (UTC), with time in 24-hour notation.

A few hours before this post, I rebooted all servers to apply kernel updates to patch the so-called “Dirty COW” privilege escalation vulnerability in the Linux kernel.  The vulnerability is indexed as CVE-2016-5195, and more information about it can be found here (with some more technical explanation here).

Due to the emergency nature of these reboots, they needed to be conducted without advance warning. I apologize for not being able to provide advance notice, and thank you for your understanding.

There is a silver lining though – since reboots needed to be performed anyway, I took advantage of them to use waiting hardware upgrades from Linode – servers minecraft1 and ridley have both now had their RAM doubled.  This only added a few minutes to the downtime the reboots would have caused otherwise.

Servers affected:

phazon.fastlizard4.org
Date and time of downtime start: In the past
Duration of downtime: Minutes
Status: Completed with no issues!
Partial list of services affected:

  • LizardWiki
  • LizardNet OTRS (emails sent to OTRS during the downtime will be delivered after the downtime concludes)
  • LizardNet Continuous Integration (Jenkins) (Gerrit will not be able to trigger any jobs during the downtime, and they will not be run after the downtime concludes)
  • LizardNet Minecraft dynamic web maps
  • LizardIRC server emerald.lizardirc.org
  • LizardIRC’s website
  • LizardMail services on phazon.fastlizard4.org (emails sent to phazon.fastlizard4.org users during the downtime will be delivered after the downtime concludes)
ridley.fastlizard4.org
Date and time of downtime start: In the past
Duration of downtime: Minutes
Status: Completed with no issues, hardware upgrades applied!
Partial list of services affected:

  • LizardWiki
  • Ladies On Two Wheels forums
  • Star Trek Games wiki
  • Wikitroid Skintest
  • LizardNet Code Review (Gerrit)
  • LizardNet Code Explorer (Gitblit)
  • LizardVPN
  • LizardNet Minecraft servers s1, c1, and c2
  • LizardNet’s Teamspeak3 server
  • Rav3nZNC
  • LizardIRC server diamond.lizardirc.org
  • LizardIRC’s website
  • LizardMail services on ridley.fastlizard4.org (emails sent to fastlizard4.org users during the downtime will be delivered after the downtime concludes)
minecraft1.fastlizard4.org
Date and time of downtime start: In the past
Duration of downtime: Minutes
Status: Completed with no issues, hardware upgrades applied!
Services affected:

25 October 2016: Emergency reboots to patch “Dirty Cow” vulnerability

7 September 2016: Scheduled reboot for critical Xen security fix

This is a past/expired downtime notification. The downtimes specified below have been completed, and remarks/results are given below as well.

Unless otherwise noted, all dates and times are given in Coordinated Universal Time (UTC), with time in 24-hour notation.

The Xen development team has released a critical and so far undisclosed Xen Security Advisory (XSA), and as such, Linode (LizardNet’s provider) will be performing emergency maintenance on all of their Xen hosts.  LizardNet’s sole Xen system, phazon.fastlizard4.org, will be rebooted as part of the endeavour to patch the Xen vulnerabilities before the public disclosure date of 8 September 2016.  (More information can be found on the Linode status blog here.)

Edit: The downtimes have been completed on schedule with no issues.  When more information becomes publicly available about the specific XSA(s) that lead to this downtime, I will update this post.

The following server and services will experience downtime:

phazon.fastlizard4.org
Date and time of downtime start: 11:00 Wednesday 7 September 2016 UTC (convert to other timezones)
Duration of downtime: Expected between 30 minutes and 1 hour, but up to 2 hours is possible
Status: Completed on schedule with no issues!
Partial list of services affected:

  • LizardWiki
  • LizardNet OTRS (emails sent to OTRS during the downtime will be delivered after the downtime concludes)
  • LizardNet Continuous Integration (Jenkins) (Gerrit will not be able to trigger any jobs during the downtime, and they will not be run after the downtime concludes)
  • LizardNet Minecraft dynamic web maps
  • LizardIRC server emerald.lizardirc.org
  • LizardIRC’s website
  • LizardMail services on phazon.fastlizard4.org (emails sent to phazon.fastlizard4.org users during the downtime will be delivered after the downtime concludes)

Apologies for the short notice on this downtime (both from me and Linode).

7 September 2016: Scheduled reboot for critical Xen security fix

22 July 2016: Scheduled reboot for critical Xen security fix

This is a past/expired downtime notification. The downtimes specified below have been completed, and remarks/results are given below as well.

Unless otherwise noted, all dates and times are given in Coordinated Universal Time (UTC), with time in 24-hour notation.

The Xen development team has released a critical and so far undisclosed Xen Security Advisory (XSA), and as such, Linode (LizardNet’s provider) will be performing emergency maintenance on all of their Xen hosts.  LizardNet’s sole Xen system, phazon.fastlizard4.org, will be rebooted as part of the endeavour to patch the Xen vulnerabilities before the public disclosure date of 26 July 2016.  (More information can be found on the Linode status blog here.)

Update: The embargoes on the Xen Seucurity Advisory that triggered this emergency scheduled reboot have been lifted, and the issue responsible seems to have been XSA-182.  Some excellent (as always) commentary about the cause and implications of this XSA has been released by the QubesOS team, and can be found here.

The following server and services will experience downtime:

phazon.fastlizard4.org
Date and time of downtime start: 11:00 Friday 22 July 2016 UTC (convert to other timezones)
Duration of downtime: Expected between 30 minutes and 1 hour, but up to 2 hours is possible
Status: Completed on schedule with no issues!
Partial list of services affected:

  • LizardWiki
  • LizardNet OTRS (emails sent to OTRS during the downtime will be delivered after the downtime concludes)
  • LizardNet Continuous Integration (Jenkins) (Gerrit will not be able to trigger any jobs during the downtime, and they will not be run after the downtime concludes)
  • LizardNet Minecraft dynamic web maps
  • LizardIRC server emerald.lizardirc.org
  • LizardIRC’s website
  • LizardMail services on phazon.fastlizard4.org (emails sent to phazon.fastlizard4.org users during the downtime will be delivered after the downtime concludes)

Apologies for the short notice on this downtime (both from me and Linode).

22 July 2016: Scheduled reboot for critical Xen security fix

9 April 2016: Scheduled maintenance downtime

This is a past/expired downtime notification. The downtimes specified below have been completed, and remarks/results are given below as well.

Unless otherwise noted, all dates and times are given in Coordinated Universal Time (UTC), with time in 24-hour notation.

Update: The scheduled maintenance has been completed with no issues.

Linode has scheduled required maintenance downtime for the server that hosts one of LizardNet’s servers, specifically, phazon.fastlizard4.org.  This downtime only affects phazon, and is expected to last about 60 minutes, though a full two hours is allocated to the downtime and may be necessary.  This downtime does not seem to be security related.

The following server and services will experience downtime:

phazon.fastlizard4.org
Date and time of downtime start: 03:00 Saturday 9 April 2016 UTC (convert to other timezones)
Duration of downtime: One hour expected, but a window of two hours has been allocated and the full two hours may be necessary.
Status: Completed with no issues.
Partial list of services affected:

  • LizardWiki
  • LizardNet OTRS (emails sent to OTRS during the downtime will be delivered after the downtime concludes)
  • LizardNet Continuous Integration (Jenkins) (Gerrit will not be able to trigger any jobs during the downtime, and they will not be run after the downtime concludes)
  • LizardNet Minecraft dynamic web maps
  • LizardIRC server emerald.lizardirc.org
  • LizardIRC’s website
  • LizardMail services on phazon.fastlizard4.org (emails sent to phazon.fastlizard4.org users during the downtime will be delivered after the downtime concludes)

Thank you in advance for your patience!

9 April 2016: Scheduled maintenance downtime

18 February 2016: Emergency reboots to fix multiple critical security issues

This is a past/expired downtime notification. The downtimes specified below have been completed, and remarks/results are given below as well.

Unless otherwise noted, all dates and times are given in Coordinated Universal Time (UTC), with time in 24-hour notation.

Update: The reboots have all been completed.

In the hours following this post, all LizardNet servers (ridley.fastlizard4.org, phazon.fastlizard4.org, and minecraft1.fastlizard4.org) will be rebooted so patches for multiple critical security vulnerabilities can be applied.  The patches include fixes for CVE-2016-0728 (Linux kernel privilege escalation) and CVE-2015-7547 (glibc getaddrinfo stack-based buffer overflow) – more information about these vulnerabilities can be found at their respective links.

Due to the emergency nature of these reboots, they will be occurring almost immediately after this post.  I apologize for not being able to provide more advance notice, and thank you for your understanding.

Servers affected:

phazon.fastlizard4.org
Date and time of downtime start: Immediately
Duration of downtime: Minutes
Status: Completed with no issues!
Partial list of services affected:

  • LizardWiki
  • LizardNet OTRS (emails sent to OTRS during the downtime will be delivered after the downtime concludes)
  • LizardNet Continuous Integration (Jenkins) (Gerrit will not be able to trigger any jobs during the downtime, and they will not be run after the downtime concludes)
  • LizardNet Minecraft dynamic web maps
  • LizardIRC server emerald.lizardirc.org
  • LizardIRC’s website
  • LizardMail services on phazon.fastlizard4.org (emails sent to phazon.fastlizard4.org users during the downtime will be delivered after the downtime concludes)
ridley.fastlizard4.org
Date and time of downtime start: Immediately
Duration of downtime: Minutes
Status: Completed with no issues!
Partial list of services affected:

  • LizardWiki
  • Star Trek Games wiki
  • Wikitroid Skintest
  • LizardNet Code Review (Gerrit)
  • LizardNet Code Explorer (Gitblit)
  • LizardVPN
  • LizardNet Minecraft servers s1, c1, and c2
  • LizardNet’s Teamspeak3 server
  • Rav3nZNC
  • LizardIRC server diamond.lizardirc.org
  • LizardIRC’s website
  • LizardMail services on ridley.fastlizard4.org (emails sent to fastlizard4.org users during the downtime will be delivered after the downtime concludes)
minecraft1.fastlizard4.org
Date and time of downtime start: Immediately
Duration of downtime: Minutes
Status: Completed with no issues!
Services affected:

18 February 2016: Emergency reboots to fix multiple critical security issues

14 December 2015: Scheduled reboot for critical Xen security fixes

This is a past/expired downtime notification. The downtimes specified below have been completed, and remarks/results are given below as well.

Unless otherwise noted, all dates and times are given in Coordinated Universal Time (UTC), with time in 24-hour notation.

Once again, the Xen development team has released several highly-critical and so far undisclosed Xen Security Advisories (XSAs), and as such, Linode (LizardNet’s provider) will be performing emergency maintenance on all of their Xen hosts.  LizardNet’s sole remaining Xen system, phazon.fastlizard4.org, will be rebooted as part of the endeavour to patch the Xen vulnerabilities before the public disclosure date of 17 December 2015.  (More information can be found on the Linode status blog here.)

Update: The Xen Security team has lifted the embargo and publicly disclosed the vulnerabilities: XSA-155, XSA-157, XSA-164, XSA-165, and XSA-166.  Of these, XSA-155 and XSA-165 appear to be by far the most serious issues.  The QubesOS folks have published a commentary addressing these bugs.

The following server and services will experience downtime:

phazon.fastlizard4.org
Date and time of downtime start: 10:00 Monday 14 December 2015 UTC (convert to other timezones)
Duration of downtime: Expected between 30 minutes and 1 hour, but up to 2 hours is possible
Status: Completed with no issues, despite late start (10:23 UTC)
Partial list of services affected:

  • LizardWiki
  • LizardNet OTRS (emails sent to OTRS during the downtime will be delivered after the downtime concludes)
  • LizardNet Continuous Integration (Jenkins) (Gerrit will not be able to trigger any jobs during the downtime, and they will not be run after the downtime concludes)
  • LizardNet Minecraft dynamic web maps
  • LizardIRC server emerald.lizardirc.org
  • LizardIRC’s website
  • LizardMail services on phazon.fastlizard4.org (emails sent to phazon.fastlizard4.org users during the downtime will be delivered after the downtime concludes)

Apologies for the short notice on this downtime (both from me and Linode).

14 December 2015: Scheduled reboot for critical Xen security fixes