Post-Mortem: ridley.fastlizard4.org downtime due to hypervisor bug

Did you know that servers are capable of detecting when a sysadmin wants to get a decent night of sleep for once?

One of LizardNet’s servers, ridley.fastlizard4.org, experienced several unscheduled reboots, with the first occurring around 12:27 UTC today (Saturday 29 July 2017).  The unscheduled reboots were followed by an extended downtime while the server was migrated to a stable host server.  Ridley was back up and operating normally by 14:20 UTC.  The first reboot was caused by a bug in the hypervisor software on the old host server, and the subsequent reboots were an attempt to diagnose some strange performance issues that surfaced in the aftermath.  The extended downtime was due to the migration of ridley to a host server where the hypervisor bug was patched.  This should resolve the unexpected reboots, and I believe it will also resolve the performance problems that were observed after the first unexpected reboot.  Everything should be back to normal now.  Thank you for your patience, and many thanks to Linode’s excellent support team for their assistance in resolving this.

The following is a partial list of services that were unavailable during this unexpected downtime:

  • LizardWiki
  • Ladies On Two Wheels forums
  • Star Trek Games wiki
  • Wikitroid Skintest
  • LizardNet Code Review (Gerrit)
  • LizardNet Code Explorer (Gitblit)
  • LizardVPN
  • LizardNet Minecraft servers s1, c1, and c2
  • LizardNet’s Teamspeak3 server
  • LizardIRC server diamond.lizardirc.org
  • LizardIRC’s website
  • LizardMail services on ridley.fastlizard4.org
Advertisements
Post-Mortem: ridley.fastlizard4.org downtime due to hypervisor bug

Post-Mortem: ridley.fastlizard4.org downtime due to hard crash

Earlier today, at 23:00:02 UTC on Wednesday 23 November 2016, ridley.fastlizard4.org suffered a hard crash resulting in a brief unexpected downtime.  The server was automatically brought back up by monitoring systems, followed by me verifying that everything is still functioning normally.  All services should be restored to normal at this time.  I have not yet identified a definitive cause for the crash; however, I will continue to analyze the data available to me and monitor for any further unexpected events.

The following is a partial list of services that were unavailable during this unexpected downtime:

  • LizardWiki
  • Ladies On Two Wheels forums
  • Star Trek Games wiki
  • Wikitroid Skintest
  • LizardNet Code Review (Gerrit)
  • LizardNet Code Explorer (Gitblit)
  • LizardVPN
  • LizardNet Minecraft servers s1, c1, and c2
  • LizardNet’s Teamspeak3 server
  • LizardIRC server diamond.lizardirc.org
  • LizardIRC’s website
  • LizardMail services on ridley.fastlizard4.org (emails sent to fastlizard4.org users during the downtime will be delivered after the downtime concludes)
Post-Mortem: ridley.fastlizard4.org downtime due to hard crash

Post-Mortem: ridley.fastlizard4.org downtime due to hardware problems

One of LizardNet’s servers, ridley.fastlizard4.org, was unexpectedly down from just after 14:00 UTC to just after 16:00 UTC today (Wednesday 3 August 2016) due to its host Linode server suffering a hardware problem.  The problem has since been fixed, and all services should now be running normally.  Thank you for your patience!

The following is a partial list of services that were unavailable during this unexpected downtime:

  • LizardWiki
  • Ladies On Two Wheels forums
  • Star Trek Games wiki
  • Wikitroid Skintest
  • LizardNet Code Review (Gerrit)
  • LizardNet Code Explorer (Gitblit)
  • LizardVPN
  • LizardNet Minecraft servers s1, c1, c2, and s2.5
  • LizardNet’s Teamspeak3 server
  • Rav3nZNC
  • LizardIRC server diamond.lizardirc.org
  • LizardIRC’s website
  • LizardMail services on ridley.fastlizard4.org
Post-Mortem: ridley.fastlizard4.org downtime due to hardware problems