This is a past/expired downtime notification. The downtimes specified below have been completed, and remarks/results are given below as well.

Unless otherwise noted, all dates and times are given in Coordinated Universal Time (UTC), with time in 24-hour notation.

Edit: Though it started just a bit late (05:07 UTC), this downtime has been completed and everything is up and running as of 05:46 UTC.  I’ll update this blog post at some point after the XSAs become public with more information about those.

Third time in three months.  It looks like this may just be a monthly thing now.

The Xen development team has released several critical and so far undisclosed Xen Security Advisories (XSAs), and as such, Linode (LizardNet’s provider) will be performing emergency maintenance on all of their Xen hosts. LizardNet’s sole Xen system, phazon.fastlizard4.org, will be rebooted as part of the endeavour to patch the Xen vulnerabilities before the public disclosure time of 12 October 2017. (More information can be found on the Linode status blog here; also, interestingly, last month’s XSAs also had a disclosure date of the 12th – September 12th specifically).

This blog post will be updated upon downtime completion, and when the relevant XSAs are made public.

The following server and services will experience downtime:

phazon.fastlizard4.org

Date and time of downtime start: 05:00 Wednesday 11 October 2017 UTC (convert to other timezones)

Duration of downtime: Expected between 30 minutes and 1 hour, but up to 2 hours is possible

Status: Completed on schedule with no issues!

Partial list of services affected:

• LizardWiki
• LizardNet OTRS (emails sent to OTRS during the downtime will be delivered after the downtime concludes)
• LizardNet Continuous Integration (Jenkins) (Gerrit will not be able to trigger any jobs during the downtime, and they will not be run after the downtime concludes)
• LizardNet Minecraft dynamic web maps
• LizardIRC server emerald.lizardirc.org
• LizardIRC’s website
• LizardMail services on phazon.fastlizard4.org (emails sent to phazon.fastlizard4.org users during the downtime will be delivered after the downtime concludes)

This is a past/expired downtime notification. The downtimes specified below have been completed, and remarks/results are given below as well.

Unless otherwise noted, all dates and times are given in Coordinated Universal Time (UTC), with time in 24-hour notation.

Edit: The downtime was completed on schedule at 05:34 UTC, and everything appears to now be back to normal.  I’ll update this blog post with more details about the patches when the XSAs go public.

Update 2: It looks like the XSAs patched in this round of reboots were XSAs 231 through 234 (you can find the full list here), with analysis by the QubesOS team.

I swear we did this last month.  Oh, right, that’s because we did.

The Xen development team has released several critical and so far undisclosed Xen Security Advisories (XSAs), and as such, Linode (LizardNet’s provider) will be performing emergency maintenance on all of their Xen hosts. LizardNet’s sole Xen system, phazon.fastlizard4.org, will be rebooted as part of the endeavour to patch the Xen vulnerabilities before the public disclosure time of 12 September 2017. (More information can be found on the Linode status blog here.)

This blog post will be updated upon downtime completion, and when the relevant XSAs are made public.

The following server and services will experience downtime:

phazon.fastlizard4.org

Date and time of downtime start: 05:00 Saturday 9 September 2017 UTC (convert to other timezones)

Duration of downtime: Expected between 30 minutes and 1 hour, but up to 2 hours is possible

Status: Completed on schedule with no issues!

Partial list of services affected:

• LizardWiki
• LizardNet OTRS (emails sent to OTRS during the downtime will be delivered after the downtime concludes)
• LizardNet Continuous Integration (Jenkins) (Gerrit will not be able to trigger any jobs during the downtime, and they will not be run after the downtime concludes)
• LizardNet Minecraft dynamic web maps
• LizardIRC server emerald.lizardirc.org
• LizardIRC’s website
• LizardMail services on phazon.fastlizard4.org (emails sent to phazon.fastlizard4.org users during the downtime will be delivered after the downtime concludes)

This is a past/expired downtime notification. The downtimes specified below have been completed, and remarks/results are given below as well.

Unless otherwise noted, all dates and times are given in Coordinated Universal Time (UTC), with time in 24-hour notation.

Update: The downtime was completed on-time and without issues at 03:49 UTC.

Update 2: It looks like the XSAs patched in this round of reboots were XSAs 226 through 230 (you can find the full list here), with analysis by the QubesOS team.

Woah.  Déjà vu, anyone?

The Xen development team has released several critical and so far undisclosed Xen Security Advisories (XSAs) (is it just me, or does it feel like this has been happening a lot recently?), and as such, Linode (LizardNet’s provider) will be performing emergency maintenance on all of their Xen hosts. LizardNet’s sole Xen system, phazon.fastlizard4.org, will be rebooted as part of the endeavour to patch the Xen vulnerabilities before the public disclosure time of 15 August 2017. (More information can be found on the Linode status blog here.)

This blog post will be updated upon downtime completion, and when the relevant XSAs are made public.

The following server and services will experience downtime:

phazon.fastlizard4.org

Date and time of downtime start: 03:00 Saturday 12 August 2017 UTC (convert to other timezones)

Duration of downtime: Expected between 30 minutes and 1 hour, but up to 2 hours is possible

Status: Completed on schedule with no issues!

Partial list of services affected:

• LizardWiki
• LizardNet OTRS (emails sent to OTRS during the downtime will be delivered after the downtime concludes)
• LizardNet Continuous Integration (Jenkins) (Gerrit will not be able to trigger any jobs during the downtime, and they will not be run after the downtime concludes)
• LizardNet Minecraft dynamic web maps
• LizardIRC server emerald.lizardirc.org
• LizardIRC’s website
• LizardMail services on phazon.fastlizard4.org (emails sent to phazon.fastlizard4.org users during the downtime will be delivered after the downtime concludes)

This is a past/expired downtime notification. The downtimes specified below have been completed, and remarks/results are given below as well.

Unless otherwise noted, all dates and times are given in Coordinated Universal Time (UTC), with time in 24-hour notation.

Update: The reboot was completed successfully automatically at 11:21 UTC.  All systems should now be back to normal.  I’ll update this blog post with more information about the XSAs when they go public.

Update 2: It looks like there was a huge list of XSAs to be patched this time around, so I’ll just hand it over to the QubesOS folks for their analysis.

Well, here we go again.  The Xen development team has released several critical and so far undisclosed Xen Security Advisories (XSAs), and as such, Linode (LizardNet’s provider) will be performing emergency maintenance on all of their Xen hosts. LizardNet’s sole Xen system, phazon.fastlizard4.org, will be rebooted as part of the endeavour to patch the Xen vulnerabilities before the public disclosure time of 20 June 2017 12:00 UTC. (More information can be found on the Linode status blog here.)

This blog post will be updated upon downtime completion, and when the relevant XSAs are made public.

The following server and services will experience downtime:

phazon.fastlizard4.org

Date and time of downtime start: 11:00 Friday 16 June 2017 UTC (convert to other timezones)

Duration of downtime: Expected between 30 minutes and 1 hour, but up to 2 hours is possible

Status: Completed on schedule with no issues!

Partial list of services affected:

• LizardWiki
• LizardNet OTRS (emails sent to OTRS during the downtime will be delivered after the downtime concludes)
• LizardNet Continuous Integration (Jenkins) (Gerrit will not be able to trigger any jobs during the downtime, and they will not be run after the downtime concludes)
• LizardNet Minecraft dynamic web maps
• LizardIRC server emerald.lizardirc.org
• LizardIRC’s website
• LizardMail services on phazon.fastlizard4.org (emails sent to phazon.fastlizard4.org users during the downtime will be delivered after the downtime concludes)

This is a past/expired downtime notification. The downtimes specified below have been completed, and remarks/results are given below as well.

Unless otherwise noted, all dates and times are given in Coordinated Universal Time (UTC), with time in 24-hour notation.

Update: The downtime was completed successfully at 11:20 UTC, and service is now back to normal.  This post will be updated again with the XSAs responsible for this downtime are made public.

Update 2: It looks like there were three XSAs responsible for this one: XSA-213, XSA-214, and XSA-215, with the former two being the most critical.  The QubesOS security writeup can be found here.

The Xen development team has released several critical and so far undisclosed Xen Security Advisories (XSAs), and as such, Linode (LizardNet’s provider) will be performing emergency maintenance on all of their Xen hosts. LizardNet’s sole Xen system, phazon.fastlizard4.org, will be rebooted as part of the endeavour to patch the Xen vulnerabilities before the public disclosure time of 2 May 2017 12:00 UTC. (More information can be found on the Linode status blog here.)

This blog post will be updated upon downtime completion, and when the relevant XSAs are made public.

The following server and services will experience downtime:

phazon.fastlizard4.org

Date and time of downtime start: 11:00 Friday 28 April 2017 UTC (convert to other timezones)

Duration of downtime: Expected between 30 minutes and 1 hour, but up to 2 hours is possible

Status: Completed on schedule with no issues!

Partial list of services affected:

• LizardWiki
• LizardNet OTRS (emails sent to OTRS during the downtime will be delivered after the downtime concludes)
• LizardNet Continuous Integration (Jenkins) (Gerrit will not be able to trigger any jobs during the downtime, and they will not be run after the downtime concludes)
• LizardNet Minecraft dynamic web maps
• LizardIRC server emerald.lizardirc.org
• LizardIRC’s website
• LizardMail services on phazon.fastlizard4.org (emails sent to phazon.fastlizard4.org users during the downtime will be delivered after the downtime concludes)

This is a past/expired downtime notification. The downtimes specified below have been completed, and remarks/results are given below as well.

Unless otherwise noted, all dates and times are given in Coordinated Universal Time (UTC), with time in 24-hour notation.

Update: The downtime is now completed.  Now that the relevant XSA is released, it looks like XSA-212 is responsible.  Once again, the QubesOS folks have put out a good summary of the XSA here.

The Xen development team has released a critical and so far undisclosed Xen Security Advisory (XSA), and as such, Linode (LizardNet’s provider) will be performing emergency maintenance on all of their Xen hosts. LizardNet’s sole Xen system, phazon.fastlizard4.org, will be rebooted as part of the endeavour to patch the Xen vulnerabilities before the public disclosure date of 4 April 2017. (More information can be found on the Linode status blog here.)

Note that this downtime is unrelated to the earlier downtime for OS upgrades on phazon.

The following server and services will experience downtime:

phazon.fastlizard4.org

Date and time of downtime start: 11:00 Tuesday 4 April 2017 UTC (convert to other timezones)

Duration of downtime: Expected between 30 minutes and 1 hour, but up to 2 hours is possible

Status: Completed on schedule with no issues!

Partial list of services affected:

• LizardWiki
• LizardNet OTRS (emails sent to OTRS during the downtime will be delivered after the downtime concludes)
• LizardNet Continuous Integration (Jenkins) (Gerrit will not be able to trigger any jobs during the downtime, and they will not be run after the downtime concludes)
• LizardNet Minecraft dynamic web maps
• LizardIRC server emerald.lizardirc.org
• LizardIRC’s website
• LizardMail services on phazon.fastlizard4.org (emails sent to phazon.fastlizard4.org users during the downtime will be delivered after the downtime concludes)

This is a past/expired downtime notification. The downtimes specified below have been completed, and remarks/results are given below as well.

Unless otherwise noted, all dates and times are given in Coordinated Universal Time (UTC), with time in 24-hour notation.

Update: The downtime was completed behind schedule at about 09:45 UTC.  Several unexpected issues arose during the downtime, one of which required me to rollback the upgrade by restoring the entire server from a backup, then reattempting the upgrade.  At this time, however, I have examined the server and I believe everything is working normally.  I apologize for this downtime taking even longer than expected.  I may delay the planned upgrade of ridley by an extra week to allow me extra time to analyze the situation on phazon to double-check that everything is working normally; if I do make such a delay, it will be announced through the normal channels.  If you use phazon or its services and notice anything amiss, please do not hesitate to contact me directly.

Server phazon.fastlizard4.org has been scheduled for extended downtime to allow for its operating system to be upgraded from Ubuntu Linux 12.04 LTS (precise) to 16.04 LTS (xenial).  10 hours is being scheduled for this downtime, which should ensure that there is enough time to handle any problems that arise.  The downtime is scheduled to begin on Sunday 26 March 2017 at 21:00 UTC.  The downtime may last significantly less time if things go better than expected, or significantly more time if more things go wrong than expected.  In addition, due to the nature of the upgrade process, services may be partially available during the downtime – however, stability cannot be guaranteed from when the downtime begins until the downtime is declared to be complete and the downtime notifications are removed from display.  This post will be updated if the downtime completes early, or if it appears that it will take longer than expected.  In addition, due to the extended nature of this downtime, it may be necessary to reschedule it before it begins – this post (and all other posted downtime notifications) will also be updated in that case.

More details about the upgrade process can be found in this post.

The following server and services will experience downtime:

phazon.fastlizard4.org

Date and time of downtime start: 21:00 Sunday 26 March 2017 UTC (convert to other timezones)

Duration of downtime: Approximately 10 hours anticipated, but expected to vary

Status: Completed with issues

Partial list of services affected:

• LizardWiki
• LizardNet OTRS (emails sent to OTRS during the downtime will be delivered after the downtime concludes)
• LizardNet Continuous Integration (Jenkins) (Gerrit will not be able to trigger any jobs during the downtime, and they will not be run after the downtime concludes)
• LizardNet Minecraft dynamic web maps
• LizardIRC server emerald.lizardirc.org
• LizardIRC’s website
• LizardMail services on phazon.fastlizard4.org (emails sent to phazon.fastlizard4.org users during the downtime will be delivered after the downtime concludes)

This is a past/expired downtime notification. The downtimes specified below have been completed, and remarks/results are given below as well.

Unless otherwise noted, all dates and times are given in Coordinated Universal Time (UTC), with time in 24-hour notation.

Update: The downtime is now completed, and everything should now be running normally again.  The upgrade went mostly smoothly, only taking about half an hour longer than expected.  Please feel free to let me know if you notice anything amiss.

Server ridley.fastlizard4.org has been scheduled for extended downtime to allow for its operating system to be upgraded from Ubuntu Linux 12.04 LTS (precise) to 16.04 LTS (xenial).  12 hours is being scheduled for this downtime, which should ensure that there is enough time to handle any problems that arise.  The downtime is scheduled to begin on Sunday 2 April 2017 at 20:00 UTC.  The downtime may last significantly less time if things go better than expected, or significantly more time if more things go wrong than expected.  In addition, due to the nature of the upgrade process, services may be partially available during the downtime – however, stability cannot be guaranteed from when the downtime begins until the downtime is declared to be complete and the downtime notifications are removed from display.  This post will be updated if the downtime completes early, or if it appears that it will take longer than expected.  In addition, due to the extended nature of this downtime, it may be necessary to reschedule it before it begins – this post (and all other posted downtime notifications) will also be updated in that case.

More details about the upgrade process can be found in this post.

The following server and services will experience downtime:

ridley.fastlizard4.org

Date and time of downtime start: 20:00 Sunday 2 April 2017 UTC (convert to other timezones)

Duration of downtime: Approximately 12 hours anticipated, but expected to vary

Status: Completed, but took about 30 minutes longer than expected (total duration 12.5 hours)

Partial list of services affected:

• LizardWiki
• Ladies On Two Wheels forums
• Star Trek Games wiki
• Wikitroid Skintest
• LizardNet Code Review (Gerrit)
• LizardNet Code Explorer (Gitblit)
• LizardVPN
• LizardNet Minecraft servers s1, c1, and c2
• LizardNet’s Teamspeak3 server
• LizardIRC server diamond.lizardirc.org
• LizardIRC’s website
• LizardMail services on ridley.fastlizard4.org (emails sent to fastlizard4.org users during the downtime will be delivered after the downtime concludes)

This is a past/expired downtime notification. The downtimes specified below have been completed, and remarks/results are given below as well.

Unless otherwise noted, all dates and times are given in Coordinated Universal Time (UTC), with time in 24-hour notation.

Server minecraft1.fastlizard4.org has been scheduled for extended downtime to allow for its operating system to be upgraded from Ubuntu Linux 12.04 LTS (precise) to 16.04 LTS (xenial).  12 hours is being scheduled for this downtime, which should ensure that there is enough time to handle any problems that arise.  The downtime is scheduled to begin on Sunday 12 March 2017 at 21:00 UTC.  The downtime may last significantly less time if things go better than expected, or significantly more time if more things go wrong than expected.  In addition, due to the nature of the upgrade process, services may be partially available during the downtime – however, stability cannot be guaranteed from when the downtime begins until the downtime is declared to be complete and the downtime notifications are removed from display.  This post will be updated if the downtime completes early, or if it appears that it will take longer than expected.  In addition, due to the extended nature of this downtime, it may be necessary to reschedule it before it begins – this post (and all other posted downtime notifications) will also be updated in that case.

More details about the upgrade process can be found in this post.

Edit: The downtime and upgrades are now complete.  A few issues cropped up in the process, such as a couple upstart jobs that failed to properly migrate over to systemd, and some unexpected config changes that I’ll need to watch out for when upgrading my remaining servers.  Everything should now be working, though!

The following server and services will experience downtime:

minecraft1.fastlizard4.org

Date and time of downtime start: 21:00 Sunday 12 March 2017 UTC (convert to other timezones)

Duration of downtime: Approximately 12 hours anticipated, but expected to vary

Status: Completed on schedule with no issues!  (Total time 5.5 hours out of 12 expected)

Services affected:

• LizardNet Minecraft servers s2, m1, and m2