20 November 2016: Scheduled reboot for critical Xen security fix

This is a past/expired downtime notification. The downtimes specified below have been completed, and remarks/results are given below as well.

Unless otherwise noted, all dates and times are given in Coordinated Universal Time (UTC), with time in 24-hour notation.

The Xen development team has released several critical and so far undisclosed Xen Security Advisories (XSAs), and as such, Linode (LizardNet’s provider) will be performing emergency maintenance on all of their Xen hosts.  LizardNet’s sole Xen system, phazon.fastlizard4.org, will be rebooted as part of the endeavour to patch the Xen vulnerabilities before the public disclosure date of 22 November 2016.  (More information can be found on the Linode status blog here.)

The following server and services will experience downtime:

phazon.fastlizard4.org
Date and time of downtime start: 12:00 Sunday 20 November 2016 UTC (convert to other timezones)
Duration of downtime: Expected between 30 minutes and 1 hour, but up to 2 hours is possible
Status: Completed on schedule with no issues!
Partial list of services affected:

  • LizardWiki
  • LizardNet OTRS (emails sent to OTRS during the downtime will be delivered after the downtime concludes)
  • LizardNet Continuous Integration (Jenkins) (Gerrit will not be able to trigger any jobs during the downtime, and they will not be run after the downtime concludes)
  • LizardNet Minecraft dynamic web maps
  • LizardIRC server emerald.lizardirc.org
  • LizardIRC’s website
  • LizardMail services on phazon.fastlizard4.org (emails sent to phazon.fastlizard4.org users during the downtime will be delivered after the downtime concludes)

Apologies for the short notice on this downtime (both from me and Linode).

Advertisements
20 November 2016: Scheduled reboot for critical Xen security fix

7 September 2016: Scheduled reboot for critical Xen security fix

This is a past/expired downtime notification. The downtimes specified below have been completed, and remarks/results are given below as well.

Unless otherwise noted, all dates and times are given in Coordinated Universal Time (UTC), with time in 24-hour notation.

The Xen development team has released a critical and so far undisclosed Xen Security Advisory (XSA), and as such, Linode (LizardNet’s provider) will be performing emergency maintenance on all of their Xen hosts.  LizardNet’s sole Xen system, phazon.fastlizard4.org, will be rebooted as part of the endeavour to patch the Xen vulnerabilities before the public disclosure date of 8 September 2016.  (More information can be found on the Linode status blog here.)

Edit: The downtimes have been completed on schedule with no issues.  When more information becomes publicly available about the specific XSA(s) that lead to this downtime, I will update this post.

The following server and services will experience downtime:

phazon.fastlizard4.org
Date and time of downtime start: 11:00 Wednesday 7 September 2016 UTC (convert to other timezones)
Duration of downtime: Expected between 30 minutes and 1 hour, but up to 2 hours is possible
Status: Completed on schedule with no issues!
Partial list of services affected:

  • LizardWiki
  • LizardNet OTRS (emails sent to OTRS during the downtime will be delivered after the downtime concludes)
  • LizardNet Continuous Integration (Jenkins) (Gerrit will not be able to trigger any jobs during the downtime, and they will not be run after the downtime concludes)
  • LizardNet Minecraft dynamic web maps
  • LizardIRC server emerald.lizardirc.org
  • LizardIRC’s website
  • LizardMail services on phazon.fastlizard4.org (emails sent to phazon.fastlizard4.org users during the downtime will be delivered after the downtime concludes)

Apologies for the short notice on this downtime (both from me and Linode).

7 September 2016: Scheduled reboot for critical Xen security fix

22 July 2016: Scheduled reboot for critical Xen security fix

This is a past/expired downtime notification. The downtimes specified below have been completed, and remarks/results are given below as well.

Unless otherwise noted, all dates and times are given in Coordinated Universal Time (UTC), with time in 24-hour notation.

The Xen development team has released a critical and so far undisclosed Xen Security Advisory (XSA), and as such, Linode (LizardNet’s provider) will be performing emergency maintenance on all of their Xen hosts.  LizardNet’s sole Xen system, phazon.fastlizard4.org, will be rebooted as part of the endeavour to patch the Xen vulnerabilities before the public disclosure date of 26 July 2016.  (More information can be found on the Linode status blog here.)

Update: The embargoes on the Xen Seucurity Advisory that triggered this emergency scheduled reboot have been lifted, and the issue responsible seems to have been XSA-182.  Some excellent (as always) commentary about the cause and implications of this XSA has been released by the QubesOS team, and can be found here.

The following server and services will experience downtime:

phazon.fastlizard4.org
Date and time of downtime start: 11:00 Friday 22 July 2016 UTC (convert to other timezones)
Duration of downtime: Expected between 30 minutes and 1 hour, but up to 2 hours is possible
Status: Completed on schedule with no issues!
Partial list of services affected:

  • LizardWiki
  • LizardNet OTRS (emails sent to OTRS during the downtime will be delivered after the downtime concludes)
  • LizardNet Continuous Integration (Jenkins) (Gerrit will not be able to trigger any jobs during the downtime, and they will not be run after the downtime concludes)
  • LizardNet Minecraft dynamic web maps
  • LizardIRC server emerald.lizardirc.org
  • LizardIRC’s website
  • LizardMail services on phazon.fastlizard4.org (emails sent to phazon.fastlizard4.org users during the downtime will be delivered after the downtime concludes)

Apologies for the short notice on this downtime (both from me and Linode).

22 July 2016: Scheduled reboot for critical Xen security fix

14 December 2015: Scheduled reboot for critical Xen security fixes

This is a past/expired downtime notification. The downtimes specified below have been completed, and remarks/results are given below as well.

Unless otherwise noted, all dates and times are given in Coordinated Universal Time (UTC), with time in 24-hour notation.

Once again, the Xen development team has released several highly-critical and so far undisclosed Xen Security Advisories (XSAs), and as such, Linode (LizardNet’s provider) will be performing emergency maintenance on all of their Xen hosts.  LizardNet’s sole remaining Xen system, phazon.fastlizard4.org, will be rebooted as part of the endeavour to patch the Xen vulnerabilities before the public disclosure date of 17 December 2015.  (More information can be found on the Linode status blog here.)

Update: The Xen Security team has lifted the embargo and publicly disclosed the vulnerabilities: XSA-155, XSA-157, XSA-164, XSA-165, and XSA-166.  Of these, XSA-155 and XSA-165 appear to be by far the most serious issues.  The QubesOS folks have published a commentary addressing these bugs.

The following server and services will experience downtime:

phazon.fastlizard4.org
Date and time of downtime start: 10:00 Monday 14 December 2015 UTC (convert to other timezones)
Duration of downtime: Expected between 30 minutes and 1 hour, but up to 2 hours is possible
Status: Completed with no issues, despite late start (10:23 UTC)
Partial list of services affected:

  • LizardWiki
  • LizardNet OTRS (emails sent to OTRS during the downtime will be delivered after the downtime concludes)
  • LizardNet Continuous Integration (Jenkins) (Gerrit will not be able to trigger any jobs during the downtime, and they will not be run after the downtime concludes)
  • LizardNet Minecraft dynamic web maps
  • LizardIRC server emerald.lizardirc.org
  • LizardIRC’s website
  • LizardMail services on phazon.fastlizard4.org (emails sent to phazon.fastlizard4.org users during the downtime will be delivered after the downtime concludes)

Apologies for the short notice on this downtime (both from me and Linode).

14 December 2015: Scheduled reboot for critical Xen security fixes

20 October 2015 – 24 October 2015: Scheduled Reboots for critical Xen security fixes and KVM upgrades

This is a past/expired downtime notification. The downtimes specified below have been completed, and remarks/results are given below as well.

Unless otherwise noted, all dates and times are given in Coordinated Universal Time (UTC), with time in 24-hour notation.

The Xen development team has released several highly-critical and so far undisclosed Xen Security Advisories (XSAs), and as such, Linode (LizardNet’s provider) will be performing emergency maintenance on all systems. The emergency maintenance does involve briefly shutting down all systems in a rolling fashion before the scheduled disclosure date of 29 October 2015. (For more information, visit Linode’s status blog here.)

Update: The XSAs have been disclosed by the Xen Security team.  The XSA in question that prompted all this seems to be XSA-148 (also known as CVE-2015-7835explanatory commentary from the QubesOS folks).

The following servers and services will experience downtime as a result of this:

phazon.fastlizard4.org
Date and time of downtime start: 06:00 Tuesday 20 October 2015 UTC (convert to other timezones)
Duration of downtime: Expected between 30 minutes and 1 hour, but up to 2 hours is possible
Status: Completed with no issues!
Partial list of services affected:

  • LizardWiki
  • LizardNet OTRS (emails sent to OTRS during the downtime will be delivered after the downtime concludes)
  • LizardNet Continuous Integration (Jenkins) (Gerrit will not be able to trigger any jobs during the downtime, and they will not be run after the downtime concludes)
  • LizardNet Minecraft dynamic web maps
  • LizardIRC server emerald.lizardirc.org
  • LizardIRC’s website
  • LizardMail services on phazon.fastlizard4.org (emails sent to phazon.fastlizard4.org users during the downtime will be delivered after the downtime concludes)

Though the following servers are affected by the XSAs, I will instead be upgrading them to KVM before their scheduled XSA patching windows. They will only experience a single downtime, however, the downtime window could be longer if the upgrade from Xen to KVM causes any problems. These servers will also, after the KVM upgrades, not be affected by future XSAs.

ridley.fastlizard4.org
Date and time of downtime start: 05:00 Wednesday 21 October 2015 UTC (convert to other timezones)
Duration of downtime: Most likely around one hour, though could be longer if complications arise during the Xen -> KVM upgrade
Status: Completed with no issues!
Partial list of services affected:

  • LizardWiki
  • Star Trek Games wiki
  • Wikitroid Skintest
  • LizardNet Code Review (Gerrit)
  • LizardNet Code Explorer (Gitblit)
  • LizardVPN
  • LizardNet Minecraft servers s1, c1, and c2
  • LizardNet’s Teamspeak3 server
  • Rav3nZNC
  • LizardIRC server diamond.lizardirc.org
  • LizardIRC’s website
  • LizardMail services on ridley.fastlizard4.org (emails sent to fastlizard4.org users during the downtime will be delivered after the downtime concludes)
minecraft1.fastlizard4.org
Date and time of downtime start: 05:00 Saturday 24 October 2015 UTC (convert to other timezones)
Duration of downtime: Most likely around one hour, though could be longer if complications arise during the Xen -> KVM upgrade
Status: Completed with no issues!
Services affected:

I apologize for the short notice on which I’m scheduling these downtimes. Though I normally aim to announce at least a week in advance, the critical nature of the XSAs requires immediate attention (indeed, I didn’t receive the reboot schedule from Linode until only shortly before this writing).

20 October 2015 – 24 October 2015: Scheduled Reboots for critical Xen security fixes and KVM upgrades