11 October 2017: Monthly (apparently) reboot for Xen Security Advisories

This is a past/expired downtime notification. The downtimes specified below have been completed, and remarks/results are given below as well.

Unless otherwise noted, all dates and times are given in Coordinated Universal Time (UTC), with time in 24-hour notation.

Edit: Though it started just a bit late (05:07 UTC), this downtime has been completed and everything is up and running as of 05:46 UTC.  I’ll update this blog post at some point after the XSAs become public with more information about those.

Third time in three months.  It looks like this may just be a monthly thing now.

The Xen development team has released several critical and so far undisclosed Xen Security Advisories (XSAs), and as such, Linode (LizardNet’s provider) will be performing emergency maintenance on all of their Xen hosts. LizardNet’s sole Xen system, phazon.fastlizard4.org, will be rebooted as part of the endeavour to patch the Xen vulnerabilities before the public disclosure time of 12 October 2017. (More information can be found on the Linode status blog here; also, interestingly, last month’s XSAs also had a disclosure date of the 12th – September 12th specifically).

This blog post will be updated upon downtime completion, and when the relevant XSAs are made public.

The following server and services will experience downtime:

phazon.fastlizard4.org
Date and time of downtime start: 05:00 Wednesday 11 October 2017 UTC (convert to other timezones)
Duration of downtime: Expected between 30 minutes and 1 hour, but up to 2 hours is possible
Status: Completed on schedule with no issues!
Partial list of services affected:

  • LizardWiki
  • LizardNet OTRS (emails sent to OTRS during the downtime will be delivered after the downtime concludes)
  • LizardNet Continuous Integration (Jenkins) (Gerrit will not be able to trigger any jobs during the downtime, and they will not be run after the downtime concludes)
  • LizardNet Minecraft dynamic web maps
  • LizardIRC server emerald.lizardirc.org
  • LizardIRC’s website
  • LizardMail services on phazon.fastlizard4.org (emails sent to phazon.fastlizard4.org users during the downtime will be delivered after the downtime concludes)
Advertisements
11 October 2017: Monthly (apparently) reboot for Xen Security Advisories

9 September 2017: More Xen Security Advisories, more problems

This is a past/expired downtime notification. The downtimes specified below have been completed, and remarks/results are given below as well.

Unless otherwise noted, all dates and times are given in Coordinated Universal Time (UTC), with time in 24-hour notation.

Edit: The downtime was completed on schedule at 05:34 UTC, and everything appears to now be back to normal.  I’ll update this blog post with more details about the patches when the XSAs go public.

Update 2: It looks like the XSAs patched in this round of reboots were XSAs 231 through 234 (you can find the full list here), with analysis by the QubesOS team.

I swear we did this last month.  Oh, right, that’s because we did.

The Xen development team has released several critical and so far undisclosed Xen Security Advisories (XSAs), and as such, Linode (LizardNet’s provider) will be performing emergency maintenance on all of their Xen hosts. LizardNet’s sole Xen system, phazon.fastlizard4.org, will be rebooted as part of the endeavour to patch the Xen vulnerabilities before the public disclosure time of 12 September 2017. (More information can be found on the Linode status blog here.)

This blog post will be updated upon downtime completion, and when the relevant XSAs are made public.

The following server and services will experience downtime:

phazon.fastlizard4.org
Date and time of downtime start: 05:00 Saturday 9 September 2017 UTC (convert to other timezones)
Duration of downtime: Expected between 30 minutes and 1 hour, but up to 2 hours is possible
Status: Completed on schedule with no issues!
Partial list of services affected:

  • LizardWiki
  • LizardNet OTRS (emails sent to OTRS during the downtime will be delivered after the downtime concludes)
  • LizardNet Continuous Integration (Jenkins) (Gerrit will not be able to trigger any jobs during the downtime, and they will not be run after the downtime concludes)
  • LizardNet Minecraft dynamic web maps
  • LizardIRC server emerald.lizardirc.org
  • LizardIRC’s website
  • LizardMail services on phazon.fastlizard4.org (emails sent to phazon.fastlizard4.org users during the downtime will be delivered after the downtime concludes)
9 September 2017: More Xen Security Advisories, more problems

12 August 2017: Reboot for patching Xen Security Advisories (XSAs)

This is a past/expired downtime notification. The downtimes specified below have been completed, and remarks/results are given below as well.

Unless otherwise noted, all dates and times are given in Coordinated Universal Time (UTC), with time in 24-hour notation.

Update: The downtime was completed on-time and without issues at 03:49 UTC.

Update 2: It looks like the XSAs patched in this round of reboots were XSAs 226 through 230 (you can find the full list here), with analysis by the QubesOS team.

Woah.  Déjà vu, anyone?

The Xen development team has released several critical and so far undisclosed Xen Security Advisories (XSAs) (is it just me, or does it feel like this has been happening a lot recently?), and as such, Linode (LizardNet’s provider) will be performing emergency maintenance on all of their Xen hosts. LizardNet’s sole Xen system, phazon.fastlizard4.org, will be rebooted as part of the endeavour to patch the Xen vulnerabilities before the public disclosure time of 15 August 2017. (More information can be found on the Linode status blog here.)

This blog post will be updated upon downtime completion, and when the relevant XSAs are made public.

The following server and services will experience downtime:

phazon.fastlizard4.org
Date and time of downtime start: 03:00 Saturday 12 August 2017 UTC (convert to other timezones)
Duration of downtime: Expected between 30 minutes and 1 hour, but up to 2 hours is possible
Status: Completed on schedule with no issues!
Partial list of services affected:

  • LizardWiki
  • LizardNet OTRS (emails sent to OTRS during the downtime will be delivered after the downtime concludes)
  • LizardNet Continuous Integration (Jenkins) (Gerrit will not be able to trigger any jobs during the downtime, and they will not be run after the downtime concludes)
  • LizardNet Minecraft dynamic web maps
  • LizardIRC server emerald.lizardirc.org
  • LizardIRC’s website
  • LizardMail services on phazon.fastlizard4.org (emails sent to phazon.fastlizard4.org users during the downtime will be delivered after the downtime concludes)
12 August 2017: Reboot for patching Xen Security Advisories (XSAs)

Post-Mortem: ridley.fastlizard4.org downtime due to hypervisor bug

Did you know that servers are capable of detecting when a sysadmin wants to get a decent night of sleep for once?

One of LizardNet’s servers, ridley.fastlizard4.org, experienced several unscheduled reboots, with the first occurring around 12:27 UTC today (Saturday 29 July 2017).  The unscheduled reboots were followed by an extended downtime while the server was migrated to a stable host server.  Ridley was back up and operating normally by 14:20 UTC.  The first reboot was caused by a bug in the hypervisor software on the old host server, and the subsequent reboots were an attempt to diagnose some strange performance issues that surfaced in the aftermath.  The extended downtime was due to the migration of ridley to a host server where the hypervisor bug was patched.  This should resolve the unexpected reboots, and I believe it will also resolve the performance problems that were observed after the first unexpected reboot.  Everything should be back to normal now.  Thank you for your patience, and many thanks to Linode’s excellent support team for their assistance in resolving this.

The following is a partial list of services that were unavailable during this unexpected downtime:

  • LizardWiki
  • Ladies On Two Wheels forums
  • Star Trek Games wiki
  • Wikitroid Skintest
  • LizardNet Code Review (Gerrit)
  • LizardNet Code Explorer (Gitblit)
  • LizardVPN
  • LizardNet Minecraft servers s1, c1, and c2
  • LizardNet’s Teamspeak3 server
  • LizardIRC server diamond.lizardirc.org
  • LizardIRC’s website
  • LizardMail services on ridley.fastlizard4.org
Post-Mortem: ridley.fastlizard4.org downtime due to hypervisor bug

16 June 2017: Reboots for Xen Security Advisories: XSA Boogaloo

This is a past/expired downtime notification. The downtimes specified below have been completed, and remarks/results are given below as well.

Unless otherwise noted, all dates and times are given in Coordinated Universal Time (UTC), with time in 24-hour notation.

Update: The reboot was completed successfully automatically at 11:21 UTC.  All systems should now be back to normal.  I’ll update this blog post with more information about the XSAs when they go public.

Update 2: It looks like there was a huge list of XSAs to be patched this time around, so I’ll just hand it over to the QubesOS folks for their analysis.

Well, here we go again.  The Xen development team has released several critical and so far undisclosed Xen Security Advisories (XSAs), and as such, Linode (LizardNet’s provider) will be performing emergency maintenance on all of their Xen hosts. LizardNet’s sole Xen system, phazon.fastlizard4.org, will be rebooted as part of the endeavour to patch the Xen vulnerabilities before the public disclosure time of 20 June 2017 12:00 UTC. (More information can be found on the Linode status blog here.)

This blog post will be updated upon downtime completion, and when the relevant XSAs are made public.

The following server and services will experience downtime:

phazon.fastlizard4.org
Date and time of downtime start: 11:00 Friday 16 June 2017 UTC (convert to other timezones)
Duration of downtime: Expected between 30 minutes and 1 hour, but up to 2 hours is possible
Status: Completed on schedule with no issues!
Partial list of services affected:

  • LizardWiki
  • LizardNet OTRS (emails sent to OTRS during the downtime will be delivered after the downtime concludes)
  • LizardNet Continuous Integration (Jenkins) (Gerrit will not be able to trigger any jobs during the downtime, and they will not be run after the downtime concludes)
  • LizardNet Minecraft dynamic web maps
  • LizardIRC server emerald.lizardirc.org
  • LizardIRC’s website
  • LizardMail services on phazon.fastlizard4.org (emails sent to phazon.fastlizard4.org users during the downtime will be delivered after the downtime concludes)
16 June 2017: Reboots for Xen Security Advisories: XSA Boogaloo

28 April 2017: (Yet another) Reboot for Xen Security Advisories

This is a past/expired downtime notification. The downtimes specified below have been completed, and remarks/results are given below as well.

Unless otherwise noted, all dates and times are given in Coordinated Universal Time (UTC), with time in 24-hour notation.

Update: The downtime was completed successfully at 11:20 UTC, and service is now back to normal.  This post will be updated again with the XSAs responsible for this downtime are made public.

Update 2: It looks like there were three XSAs responsible for this one: XSA-213, XSA-214, and XSA-215, with the former two being the most critical.  The QubesOS security writeup can be found here.

The Xen development team has released several critical and so far undisclosed Xen Security Advisories (XSAs), and as such, Linode (LizardNet’s provider) will be performing emergency maintenance on all of their Xen hosts. LizardNet’s sole Xen system, phazon.fastlizard4.org, will be rebooted as part of the endeavour to patch the Xen vulnerabilities before the public disclosure time of 2 May 2017 12:00 UTC. (More information can be found on the Linode status blog here.)

This blog post will be updated upon downtime completion, and when the relevant XSAs are made public.

The following server and services will experience downtime:

phazon.fastlizard4.org
Date and time of downtime start: 11:00 Friday 28 April 2017 UTC (convert to other timezones)
Duration of downtime: Expected between 30 minutes and 1 hour, but up to 2 hours is possible
Status: Completed on schedule with no issues!
Partial list of services affected:

  • LizardWiki
  • LizardNet OTRS (emails sent to OTRS during the downtime will be delivered after the downtime concludes)
  • LizardNet Continuous Integration (Jenkins) (Gerrit will not be able to trigger any jobs during the downtime, and they will not be run after the downtime concludes)
  • LizardNet Minecraft dynamic web maps
  • LizardIRC server emerald.lizardirc.org
  • LizardIRC’s website
  • LizardMail services on phazon.fastlizard4.org (emails sent to phazon.fastlizard4.org users during the downtime will be delivered after the downtime concludes)
28 April 2017: (Yet another) Reboot for Xen Security Advisories

4 April 2017: Scheduled reboot for critical Xen security fix

This is a past/expired downtime notification. The downtimes specified below have been completed, and remarks/results are given below as well.

Unless otherwise noted, all dates and times are given in Coordinated Universal Time (UTC), with time in 24-hour notation.

Update: The downtime is now completed.  Now that the relevant XSA is released, it looks like XSA-212 is responsible.  Once again, the QubesOS folks have put out a good summary of the XSA here.

The Xen development team has released a critical and so far undisclosed Xen Security Advisory (XSA), and as such, Linode (LizardNet’s provider) will be performing emergency maintenance on all of their Xen hosts. LizardNet’s sole Xen system, phazon.fastlizard4.org, will be rebooted as part of the endeavour to patch the Xen vulnerabilities before the public disclosure date of 4 April 2017. (More information can be found on the Linode status blog here.)

Note that this downtime is unrelated to the earlier downtime for OS upgrades on phazon.

The following server and services will experience downtime:

phazon.fastlizard4.org
Date and time of downtime start: 11:00 Tuesday 4 April 2017 UTC (convert to other timezones)
Duration of downtime: Expected between 30 minutes and 1 hour, but up to 2 hours is possible
Status: Completed on schedule with no issues!
Partial list of services affected:

  • LizardWiki
  • LizardNet OTRS (emails sent to OTRS during the downtime will be delivered after the downtime concludes)
  • LizardNet Continuous Integration (Jenkins) (Gerrit will not be able to trigger any jobs during the downtime, and they will not be run after the downtime concludes)
  • LizardNet Minecraft dynamic web maps
  • LizardIRC server emerald.lizardirc.org
  • LizardIRC’s website
  • LizardMail services on phazon.fastlizard4.org (emails sent to phazon.fastlizard4.org users during the downtime will be delivered after the downtime concludes)
4 April 2017: Scheduled reboot for critical Xen security fix