26 March 2017: Operating system upgrade on phazon

This is a past/expired downtime notification. The downtimes specified below have been completed, and remarks/results are given below as well.

Unless otherwise noted, all dates and times are given in Coordinated Universal Time (UTC), with time in 24-hour notation.

Update: The downtime was completed behind schedule at about 09:45 UTC.  Several unexpected issues arose during the downtime, one of which required me to rollback the upgrade by restoring the entire server from a backup, then reattempting the upgrade.  At this time, however, I have examined the server and I believe everything is working normally.  I apologize for this downtime taking even longer than expected.  I may delay the planned upgrade of ridley by an extra week to allow me extra time to analyze the situation on phazon to double-check that everything is working normally; if I do make such a delay, it will be announced through the normal channels.  If you use phazon or its services and notice anything amiss, please do not hesitate to contact me directly.

Server phazon.fastlizard4.org has been scheduled for extended downtime to allow for its operating system to be upgraded from Ubuntu Linux 12.04 LTS (precise) to 16.04 LTS (xenial).  10 hours is being scheduled for this downtime, which should ensure that there is enough time to handle any problems that arise.  The downtime is scheduled to begin on Sunday 26 March 2017 at 21:00 UTC.  The downtime may last significantly less time if things go better than expected, or significantly more time if more things go wrong than expected.  In addition, due to the nature of the upgrade process, services may be partially available during the downtime – however, stability cannot be guaranteed from when the downtime begins until the downtime is declared to be complete and the downtime notifications are removed from display.  This post will be updated if the downtime completes early, or if it appears that it will take longer than expected.  In addition, due to the extended nature of this downtime, it may be necessary to reschedule it before it begins – this post (and all other posted downtime notifications) will also be updated in that case.

More details about the upgrade process can be found in this post.

The following server and services will experience downtime:

phazon.fastlizard4.org
Date and time of downtime start: 21:00 Sunday 26 March 2017 UTC (convert to other timezones)
Duration of downtime: Approximately 10 hours anticipated, but expected to vary
Status: Completed with issues
Partial list of services affected:

  • LizardWiki
  • LizardNet OTRS (emails sent to OTRS during the downtime will be delivered after the downtime concludes)
  • LizardNet Continuous Integration (Jenkins) (Gerrit will not be able to trigger any jobs during the downtime, and they will not be run after the downtime concludes)
  • LizardNet Minecraft dynamic web maps
  • LizardIRC server emerald.lizardirc.org
  • LizardIRC’s website
  • LizardMail services on phazon.fastlizard4.org (emails sent to phazon.fastlizard4.org users during the downtime will be delivered after the downtime concludes)
Advertisements
26 March 2017: Operating system upgrade on phazon

2 April 2017: Operating system upgrade on ridley

This is a past/expired downtime notification. The downtimes specified below have been completed, and remarks/results are given below as well.

Unless otherwise noted, all dates and times are given in Coordinated Universal Time (UTC), with time in 24-hour notation.

Update: The downtime is now completed, and everything should now be running normally again.  The upgrade went mostly smoothly, only taking about half an hour longer than expected.  Please feel free to let me know if you notice anything amiss.

Server ridley.fastlizard4.org has been scheduled for extended downtime to allow for its operating system to be upgraded from Ubuntu Linux 12.04 LTS (precise) to 16.04 LTS (xenial).  12 hours is being scheduled for this downtime, which should ensure that there is enough time to handle any problems that arise.  The downtime is scheduled to begin on Sunday 2 April 2017 at 20:00 UTC.  The downtime may last significantly less time if things go better than expected, or significantly more time if more things go wrong than expected.  In addition, due to the nature of the upgrade process, services may be partially available during the downtime – however, stability cannot be guaranteed from when the downtime begins until the downtime is declared to be complete and the downtime notifications are removed from display.  This post will be updated if the downtime completes early, or if it appears that it will take longer than expected.  In addition, due to the extended nature of this downtime, it may be necessary to reschedule it before it begins – this post (and all other posted downtime notifications) will also be updated in that case.

More details about the upgrade process can be found in this post.

The following server and services will experience downtime:

ridley.fastlizard4.org
Date and time of downtime start: 20:00 Sunday 2 April 2017 UTC (convert to other timezones)
Duration of downtime: Approximately 12 hours anticipated, but expected to vary
Status: Completed, but took about 30 minutes longer than expected (total duration 12.5 hours)
Partial list of services affected:

  • LizardWiki
  • Ladies On Two Wheels forums
  • Star Trek Games wiki
  • Wikitroid Skintest
  • LizardNet Code Review (Gerrit)
  • LizardNet Code Explorer (Gitblit)
  • LizardVPN
  • LizardNet Minecraft servers s1, c1, and c2
  • LizardNet’s Teamspeak3 server
  • LizardIRC server diamond.lizardirc.org
  • LizardIRC’s website
  • LizardMail services on ridley.fastlizard4.org (emails sent to fastlizard4.org users during the downtime will be delivered after the downtime concludes)
2 April 2017: Operating system upgrade on ridley

12 March 2017: Operating system upgrade on minecraft1

This is a past/expired downtime notification. The downtimes specified below have been completed, and remarks/results are given below as well.

Unless otherwise noted, all dates and times are given in Coordinated Universal Time (UTC), with time in 24-hour notation.

Server minecraft1.fastlizard4.org has been scheduled for extended downtime to allow for its operating system to be upgraded from Ubuntu Linux 12.04 LTS (precise) to 16.04 LTS (xenial).  12 hours is being scheduled for this downtime, which should ensure that there is enough time to handle any problems that arise.  The downtime is scheduled to begin on Sunday 12 March 2017 at 21:00 UTC.  The downtime may last significantly less time if things go better than expected, or significantly more time if more things go wrong than expected.  In addition, due to the nature of the upgrade process, services may be partially available during the downtime – however, stability cannot be guaranteed from when the downtime begins until the downtime is declared to be complete and the downtime notifications are removed from display.  This post will be updated if the downtime completes early, or if it appears that it will take longer than expected.  In addition, due to the extended nature of this downtime, it may be necessary to reschedule it before it begins – this post (and all other posted downtime notifications) will also be updated in that case.

More details about the upgrade process can be found in this post.

Edit: The downtime and upgrades are now complete.  A few issues cropped up in the process, such as a couple upstart jobs that failed to properly migrate over to systemd, and some unexpected config changes that I’ll need to watch out for when upgrading my remaining servers.  Everything should now be working, though!

The following server and services will experience downtime:

minecraft1.fastlizard4.org
Date and time of downtime start: 21:00 Sunday 12 March 2017 UTC (convert to other timezones)
Duration of downtime: Approximately 12 hours anticipated, but expected to vary
Status: Completed on schedule with no issues!  (Total time 5.5 hours out of 12 expected)
Services affected:

12 March 2017: Operating system upgrade on minecraft1

All Servers: Ubuntu Distribution Upgrades

With the upcoming end-of-life for Ubuntu Linux 12.04 LTS, I will soon be upgrading the installed Ubuntu version on all LizardNet servers, which currently all run 12.04.

The plan is to perform a double upgrade on all three servers, first from 12.04 LTS (precise) to 14.04 LTS (trusty), then from 14.04 LTS to 16.04 LTS (xenial), which is the newest LTS (long-term-support) release of Ubuntu Linux available and which is anticipated to be supported until April 2021.[1]

Servers will be upgraded in sequence, starting with minecraft1, followed by phazon, then ridley.  Before each upgrade, a full backup of the system being upgraded will be taken to allow for a rollback in case something goes seriously wrong, and once all upgrades on a system are complete I will make a detailed analysis to ensure that everything is working as expected.

As with any OS upgrade, there will be major changes that may require reconfiguration of software and services, and these will only be compounded by the fact that I am upgrading two major releases in one go.  Among the changes that I know will be problematic is the significantly different configuration format for the Apache web server introduced in version 2.4 – upgrading the OS will also entail upgrading from Apache 2.2 to Apache 2.4 then updating the existing configuration to be compatible with the new version of Apache.  Certain software may also be broken in the upgrade process, especially older software.  Such changes will increase the time required to complete the upgrade beyond the time needed to simply perform the upgrade to the operating system components themselves.

Because of this, I am initially allocating a downtime period of 12 hours for each server.  The actual time the server and the services it hosts will be unavailable may and will probably vary; for example, during the process of downloading the new operating system files, certain parts of the server will continue operating, and the downtime period may be shorter if things go better than planned, or may be longer if unexpected difficulties arise.  As minecraft1 is the first server I plan to upgrade, I will use the upgrade experience on that server to inform my estimates for the remaining two servers, and will adjust the planned downtime periods accordingly if needed.

Because of the extended downtime required by OS upgrades, certain services on servers scheduled for downtime may be relocated to other servers during the scheduled downtime period to ensure their availability; these relocations, if any, will be detailed in each servers’ downtime notification.

Finally, because of the anticipated extended nature of the downtimes, the downtime periods are not set in stone and may change if necessary to ensure that I have enough time to devote to the upgrade task.  However, I will do my best to ensure a minimum of 24 hours notice before the start of a downtime, and that all changes occur at a minimum of 24 hours before the original start of the downtime.

Downtime Schedule

This post will be updated with the scheduled downtimes for all servers as they are scheduled, and will also be updated as the upgrades are complete.

Notes

  1. “Anticipated” because the actual end-of-life date has not yet been formally announced by Canonical; however, for Long-Term-Support releases, the EOL is generally five years from the month and year of release.
All Servers: Ubuntu Distribution Upgrades

Post-Mortem: ridley.fastlizard4.org downtime due to hard crash

Earlier today, at 23:00:02 UTC on Wednesday 23 November 2016, ridley.fastlizard4.org suffered a hard crash resulting in a brief unexpected downtime.  The server was automatically brought back up by monitoring systems, followed by me verifying that everything is still functioning normally.  All services should be restored to normal at this time.  I have not yet identified a definitive cause for the crash; however, I will continue to analyze the data available to me and monitor for any further unexpected events.

The following is a partial list of services that were unavailable during this unexpected downtime:

  • LizardWiki
  • Ladies On Two Wheels forums
  • Star Trek Games wiki
  • Wikitroid Skintest
  • LizardNet Code Review (Gerrit)
  • LizardNet Code Explorer (Gitblit)
  • LizardVPN
  • LizardNet Minecraft servers s1, c1, and c2
  • LizardNet’s Teamspeak3 server
  • LizardIRC server diamond.lizardirc.org
  • LizardIRC’s website
  • LizardMail services on ridley.fastlizard4.org (emails sent to fastlizard4.org users during the downtime will be delivered after the downtime concludes)
Post-Mortem: ridley.fastlizard4.org downtime due to hard crash

20 November 2016: Scheduled reboot for critical Xen security fix

This is a past/expired downtime notification. The downtimes specified below have been completed, and remarks/results are given below as well.

Unless otherwise noted, all dates and times are given in Coordinated Universal Time (UTC), with time in 24-hour notation.

The Xen development team has released several critical and so far undisclosed Xen Security Advisories (XSAs), and as such, Linode (LizardNet’s provider) will be performing emergency maintenance on all of their Xen hosts.  LizardNet’s sole Xen system, phazon.fastlizard4.org, will be rebooted as part of the endeavour to patch the Xen vulnerabilities before the public disclosure date of 22 November 2016.  (More information can be found on the Linode status blog here.)

The following server and services will experience downtime:

phazon.fastlizard4.org
Date and time of downtime start: 12:00 Sunday 20 November 2016 UTC (convert to other timezones)
Duration of downtime: Expected between 30 minutes and 1 hour, but up to 2 hours is possible
Status: Completed on schedule with no issues!
Partial list of services affected:

  • LizardWiki
  • LizardNet OTRS (emails sent to OTRS during the downtime will be delivered after the downtime concludes)
  • LizardNet Continuous Integration (Jenkins) (Gerrit will not be able to trigger any jobs during the downtime, and they will not be run after the downtime concludes)
  • LizardNet Minecraft dynamic web maps
  • LizardIRC server emerald.lizardirc.org
  • LizardIRC’s website
  • LizardMail services on phazon.fastlizard4.org (emails sent to phazon.fastlizard4.org users during the downtime will be delivered after the downtime concludes)

Apologies for the short notice on this downtime (both from me and Linode).

20 November 2016: Scheduled reboot for critical Xen security fix

25 October 2016: Emergency reboots to patch “Dirty Cow” vulnerability

This is a past/expired downtime notification. The downtimes specified below have been completed, and remarks/results are given below as well.

Unless otherwise noted, all dates and times are given in Coordinated Universal Time (UTC), with time in 24-hour notation.

A few hours before this post, I rebooted all servers to apply kernel updates to patch the so-called “Dirty COW” privilege escalation vulnerability in the Linux kernel.  The vulnerability is indexed as CVE-2016-5195, and more information about it can be found here (with some more technical explanation here).

Due to the emergency nature of these reboots, they needed to be conducted without advance warning. I apologize for not being able to provide advance notice, and thank you for your understanding.

There is a silver lining though – since reboots needed to be performed anyway, I took advantage of them to use waiting hardware upgrades from Linode – servers minecraft1 and ridley have both now had their RAM doubled.  This only added a few minutes to the downtime the reboots would have caused otherwise.

Servers affected:

phazon.fastlizard4.org
Date and time of downtime start: In the past
Duration of downtime: Minutes
Status: Completed with no issues!
Partial list of services affected:

  • LizardWiki
  • LizardNet OTRS (emails sent to OTRS during the downtime will be delivered after the downtime concludes)
  • LizardNet Continuous Integration (Jenkins) (Gerrit will not be able to trigger any jobs during the downtime, and they will not be run after the downtime concludes)
  • LizardNet Minecraft dynamic web maps
  • LizardIRC server emerald.lizardirc.org
  • LizardIRC’s website
  • LizardMail services on phazon.fastlizard4.org (emails sent to phazon.fastlizard4.org users during the downtime will be delivered after the downtime concludes)
ridley.fastlizard4.org
Date and time of downtime start: In the past
Duration of downtime: Minutes
Status: Completed with no issues, hardware upgrades applied!
Partial list of services affected:

  • LizardWiki
  • Ladies On Two Wheels forums
  • Star Trek Games wiki
  • Wikitroid Skintest
  • LizardNet Code Review (Gerrit)
  • LizardNet Code Explorer (Gitblit)
  • LizardVPN
  • LizardNet Minecraft servers s1, c1, and c2
  • LizardNet’s Teamspeak3 server
  • Rav3nZNC
  • LizardIRC server diamond.lizardirc.org
  • LizardIRC’s website
  • LizardMail services on ridley.fastlizard4.org (emails sent to fastlizard4.org users during the downtime will be delivered after the downtime concludes)
minecraft1.fastlizard4.org
Date and time of downtime start: In the past
Duration of downtime: Minutes
Status: Completed with no issues, hardware upgrades applied!
Services affected:

25 October 2016: Emergency reboots to patch “Dirty Cow” vulnerability