2 April 2017: Operating system upgrade on ridley

This is a past/expired downtime notification. The downtimes specified below have been completed, and remarks/results are given below as well.

Unless otherwise noted, all dates and times are given in Coordinated Universal Time (UTC), with time in 24-hour notation.

Update: The downtime is now completed, and everything should now be running normally again.  The upgrade went mostly smoothly, only taking about half an hour longer than expected.  Please feel free to let me know if you notice anything amiss.

Server ridley.fastlizard4.org has been scheduled for extended downtime to allow for its operating system to be upgraded from Ubuntu Linux 12.04 LTS (precise) to 16.04 LTS (xenial).  12 hours is being scheduled for this downtime, which should ensure that there is enough time to handle any problems that arise.  The downtime is scheduled to begin on Sunday 2 April 2017 at 20:00 UTC.  The downtime may last significantly less time if things go better than expected, or significantly more time if more things go wrong than expected.  In addition, due to the nature of the upgrade process, services may be partially available during the downtime – however, stability cannot be guaranteed from when the downtime begins until the downtime is declared to be complete and the downtime notifications are removed from display.  This post will be updated if the downtime completes early, or if it appears that it will take longer than expected.  In addition, due to the extended nature of this downtime, it may be necessary to reschedule it before it begins – this post (and all other posted downtime notifications) will also be updated in that case.

More details about the upgrade process can be found in this post.

The following server and services will experience downtime:

ridley.fastlizard4.org
Date and time of downtime start: 20:00 Sunday 2 April 2017 UTC (convert to other timezones)
Duration of downtime: Approximately 12 hours anticipated, but expected to vary
Status: Completed, but took about 30 minutes longer than expected (total duration 12.5 hours)
Partial list of services affected:

  • LizardWiki
  • Ladies On Two Wheels forums
  • Star Trek Games wiki
  • Wikitroid Skintest
  • LizardNet Code Review (Gerrit)
  • LizardNet Code Explorer (Gitblit)
  • LizardVPN
  • LizardNet Minecraft servers s1, c1, and c2
  • LizardNet’s Teamspeak3 server
  • LizardIRC server diamond.lizardirc.org
  • LizardIRC’s website
  • LizardMail services on ridley.fastlizard4.org (emails sent to fastlizard4.org users during the downtime will be delivered after the downtime concludes)
Advertisements
2 April 2017: Operating system upgrade on ridley

12 March 2017: Operating system upgrade on minecraft1

This is a past/expired downtime notification. The downtimes specified below have been completed, and remarks/results are given below as well.

Unless otherwise noted, all dates and times are given in Coordinated Universal Time (UTC), with time in 24-hour notation.

Server minecraft1.fastlizard4.org has been scheduled for extended downtime to allow for its operating system to be upgraded from Ubuntu Linux 12.04 LTS (precise) to 16.04 LTS (xenial).  12 hours is being scheduled for this downtime, which should ensure that there is enough time to handle any problems that arise.  The downtime is scheduled to begin on Sunday 12 March 2017 at 21:00 UTC.  The downtime may last significantly less time if things go better than expected, or significantly more time if more things go wrong than expected.  In addition, due to the nature of the upgrade process, services may be partially available during the downtime – however, stability cannot be guaranteed from when the downtime begins until the downtime is declared to be complete and the downtime notifications are removed from display.  This post will be updated if the downtime completes early, or if it appears that it will take longer than expected.  In addition, due to the extended nature of this downtime, it may be necessary to reschedule it before it begins – this post (and all other posted downtime notifications) will also be updated in that case.

More details about the upgrade process can be found in this post.

Edit: The downtime and upgrades are now complete.  A few issues cropped up in the process, such as a couple upstart jobs that failed to properly migrate over to systemd, and some unexpected config changes that I’ll need to watch out for when upgrading my remaining servers.  Everything should now be working, though!

The following server and services will experience downtime:

minecraft1.fastlizard4.org
Date and time of downtime start: 21:00 Sunday 12 March 2017 UTC (convert to other timezones)
Duration of downtime: Approximately 12 hours anticipated, but expected to vary
Status: Completed on schedule with no issues!  (Total time 5.5 hours out of 12 expected)
Services affected:

12 March 2017: Operating system upgrade on minecraft1

20 November 2016: Scheduled reboot for critical Xen security fix

This is a past/expired downtime notification. The downtimes specified below have been completed, and remarks/results are given below as well.

Unless otherwise noted, all dates and times are given in Coordinated Universal Time (UTC), with time in 24-hour notation.

The Xen development team has released several critical and so far undisclosed Xen Security Advisories (XSAs), and as such, Linode (LizardNet’s provider) will be performing emergency maintenance on all of their Xen hosts.  LizardNet’s sole Xen system, phazon.fastlizard4.org, will be rebooted as part of the endeavour to patch the Xen vulnerabilities before the public disclosure date of 22 November 2016.  (More information can be found on the Linode status blog here.)

The following server and services will experience downtime:

phazon.fastlizard4.org
Date and time of downtime start: 12:00 Sunday 20 November 2016 UTC (convert to other timezones)
Duration of downtime: Expected between 30 minutes and 1 hour, but up to 2 hours is possible
Status: Completed on schedule with no issues!
Partial list of services affected:

  • LizardWiki
  • LizardNet OTRS (emails sent to OTRS during the downtime will be delivered after the downtime concludes)
  • LizardNet Continuous Integration (Jenkins) (Gerrit will not be able to trigger any jobs during the downtime, and they will not be run after the downtime concludes)
  • LizardNet Minecraft dynamic web maps
  • LizardIRC server emerald.lizardirc.org
  • LizardIRC’s website
  • LizardMail services on phazon.fastlizard4.org (emails sent to phazon.fastlizard4.org users during the downtime will be delivered after the downtime concludes)

Apologies for the short notice on this downtime (both from me and Linode).

20 November 2016: Scheduled reboot for critical Xen security fix

25 October 2016: Emergency reboots to patch “Dirty Cow” vulnerability

This is a past/expired downtime notification. The downtimes specified below have been completed, and remarks/results are given below as well.

Unless otherwise noted, all dates and times are given in Coordinated Universal Time (UTC), with time in 24-hour notation.

A few hours before this post, I rebooted all servers to apply kernel updates to patch the so-called “Dirty COW” privilege escalation vulnerability in the Linux kernel.  The vulnerability is indexed as CVE-2016-5195, and more information about it can be found here (with some more technical explanation here).

Due to the emergency nature of these reboots, they needed to be conducted without advance warning. I apologize for not being able to provide advance notice, and thank you for your understanding.

There is a silver lining though – since reboots needed to be performed anyway, I took advantage of them to use waiting hardware upgrades from Linode – servers minecraft1 and ridley have both now had their RAM doubled.  This only added a few minutes to the downtime the reboots would have caused otherwise.

Servers affected:

phazon.fastlizard4.org
Date and time of downtime start: In the past
Duration of downtime: Minutes
Status: Completed with no issues!
Partial list of services affected:

  • LizardWiki
  • LizardNet OTRS (emails sent to OTRS during the downtime will be delivered after the downtime concludes)
  • LizardNet Continuous Integration (Jenkins) (Gerrit will not be able to trigger any jobs during the downtime, and they will not be run after the downtime concludes)
  • LizardNet Minecraft dynamic web maps
  • LizardIRC server emerald.lizardirc.org
  • LizardIRC’s website
  • LizardMail services on phazon.fastlizard4.org (emails sent to phazon.fastlizard4.org users during the downtime will be delivered after the downtime concludes)
ridley.fastlizard4.org
Date and time of downtime start: In the past
Duration of downtime: Minutes
Status: Completed with no issues, hardware upgrades applied!
Partial list of services affected:

  • LizardWiki
  • Ladies On Two Wheels forums
  • Star Trek Games wiki
  • Wikitroid Skintest
  • LizardNet Code Review (Gerrit)
  • LizardNet Code Explorer (Gitblit)
  • LizardVPN
  • LizardNet Minecraft servers s1, c1, and c2
  • LizardNet’s Teamspeak3 server
  • Rav3nZNC
  • LizardIRC server diamond.lizardirc.org
  • LizardIRC’s website
  • LizardMail services on ridley.fastlizard4.org (emails sent to fastlizard4.org users during the downtime will be delivered after the downtime concludes)
minecraft1.fastlizard4.org
Date and time of downtime start: In the past
Duration of downtime: Minutes
Status: Completed with no issues, hardware upgrades applied!
Services affected:

25 October 2016: Emergency reboots to patch “Dirty Cow” vulnerability

7 September 2016: Scheduled reboot for critical Xen security fix

This is a past/expired downtime notification. The downtimes specified below have been completed, and remarks/results are given below as well.

Unless otherwise noted, all dates and times are given in Coordinated Universal Time (UTC), with time in 24-hour notation.

The Xen development team has released a critical and so far undisclosed Xen Security Advisory (XSA), and as such, Linode (LizardNet’s provider) will be performing emergency maintenance on all of their Xen hosts.  LizardNet’s sole Xen system, phazon.fastlizard4.org, will be rebooted as part of the endeavour to patch the Xen vulnerabilities before the public disclosure date of 8 September 2016.  (More information can be found on the Linode status blog here.)

Edit: The downtimes have been completed on schedule with no issues.  When more information becomes publicly available about the specific XSA(s) that lead to this downtime, I will update this post.

The following server and services will experience downtime:

phazon.fastlizard4.org
Date and time of downtime start: 11:00 Wednesday 7 September 2016 UTC (convert to other timezones)
Duration of downtime: Expected between 30 minutes and 1 hour, but up to 2 hours is possible
Status: Completed on schedule with no issues!
Partial list of services affected:

  • LizardWiki
  • LizardNet OTRS (emails sent to OTRS during the downtime will be delivered after the downtime concludes)
  • LizardNet Continuous Integration (Jenkins) (Gerrit will not be able to trigger any jobs during the downtime, and they will not be run after the downtime concludes)
  • LizardNet Minecraft dynamic web maps
  • LizardIRC server emerald.lizardirc.org
  • LizardIRC’s website
  • LizardMail services on phazon.fastlizard4.org (emails sent to phazon.fastlizard4.org users during the downtime will be delivered after the downtime concludes)

Apologies for the short notice on this downtime (both from me and Linode).

7 September 2016: Scheduled reboot for critical Xen security fix

22 July 2016: Scheduled reboot for critical Xen security fix

This is a past/expired downtime notification. The downtimes specified below have been completed, and remarks/results are given below as well.

Unless otherwise noted, all dates and times are given in Coordinated Universal Time (UTC), with time in 24-hour notation.

The Xen development team has released a critical and so far undisclosed Xen Security Advisory (XSA), and as such, Linode (LizardNet’s provider) will be performing emergency maintenance on all of their Xen hosts.  LizardNet’s sole Xen system, phazon.fastlizard4.org, will be rebooted as part of the endeavour to patch the Xen vulnerabilities before the public disclosure date of 26 July 2016.  (More information can be found on the Linode status blog here.)

Update: The embargoes on the Xen Seucurity Advisory that triggered this emergency scheduled reboot have been lifted, and the issue responsible seems to have been XSA-182.  Some excellent (as always) commentary about the cause and implications of this XSA has been released by the QubesOS team, and can be found here.

The following server and services will experience downtime:

phazon.fastlizard4.org
Date and time of downtime start: 11:00 Friday 22 July 2016 UTC (convert to other timezones)
Duration of downtime: Expected between 30 minutes and 1 hour, but up to 2 hours is possible
Status: Completed on schedule with no issues!
Partial list of services affected:

  • LizardWiki
  • LizardNet OTRS (emails sent to OTRS during the downtime will be delivered after the downtime concludes)
  • LizardNet Continuous Integration (Jenkins) (Gerrit will not be able to trigger any jobs during the downtime, and they will not be run after the downtime concludes)
  • LizardNet Minecraft dynamic web maps
  • LizardIRC server emerald.lizardirc.org
  • LizardIRC’s website
  • LizardMail services on phazon.fastlizard4.org (emails sent to phazon.fastlizard4.org users during the downtime will be delivered after the downtime concludes)

Apologies for the short notice on this downtime (both from me and Linode).

22 July 2016: Scheduled reboot for critical Xen security fix

9 April 2016: Scheduled maintenance downtime

This is a past/expired downtime notification. The downtimes specified below have been completed, and remarks/results are given below as well.

Unless otherwise noted, all dates and times are given in Coordinated Universal Time (UTC), with time in 24-hour notation.

Update: The scheduled maintenance has been completed with no issues.

Linode has scheduled required maintenance downtime for the server that hosts one of LizardNet’s servers, specifically, phazon.fastlizard4.org.  This downtime only affects phazon, and is expected to last about 60 minutes, though a full two hours is allocated to the downtime and may be necessary.  This downtime does not seem to be security related.

The following server and services will experience downtime:

phazon.fastlizard4.org
Date and time of downtime start: 03:00 Saturday 9 April 2016 UTC (convert to other timezones)
Duration of downtime: One hour expected, but a window of two hours has been allocated and the full two hours may be necessary.
Status: Completed with no issues.
Partial list of services affected:

  • LizardWiki
  • LizardNet OTRS (emails sent to OTRS during the downtime will be delivered after the downtime concludes)
  • LizardNet Continuous Integration (Jenkins) (Gerrit will not be able to trigger any jobs during the downtime, and they will not be run after the downtime concludes)
  • LizardNet Minecraft dynamic web maps
  • LizardIRC server emerald.lizardirc.org
  • LizardIRC’s website
  • LizardMail services on phazon.fastlizard4.org (emails sent to phazon.fastlizard4.org users during the downtime will be delivered after the downtime concludes)

Thank you in advance for your patience!

9 April 2016: Scheduled maintenance downtime