Server News, Updates, and Downtime

[Zok]

I just figured that there are already several threads like this - I'll just lump them into this one for now on. I'm keeping the thread closed so that people don't have to wade through pages of replies in order to figure out what's going on. If you've got a question or comment about anything mentioned in this thread, please email me or send me a PM.

[Zok]

Yesterday afternoon (Mar. 11, 2010) several users experienced database errors as they tried to access pages. The reason for this was that a few slow queries ate up mysql's available threads.

I identified the slow query and corrected the problem. I also added a script that will automatically kill slow queries if they use more than a minute of CPU time. Because of this, you may get an error if you run big searches during peak hours.

[Zok]

March 14 - Email server seems to be working. I switched from using the default sendmail to postfix with sasl authentication. Registration should work properly also.

[Zok]

Sorry for the brief downtime there - had to change some sql settings to prevent DB errors. Things seem to be running well now.

[Zok]

Sorry for the downtime this evening. The problem was not on zoklet's end, but rather the datacenter that we're hosted at (or an important node along the way). Zoklet's server is in montana and every traceroute I ran died in chicago. Not sure what the problem was, but we're back online

[Zok]

Things have been going slow today. It looks like it's networking - not our actual server. Unfortunately I can't do anything to solve this problem. We'll just have to make due with it being slow.

Seems to be a bit better now.

[Zok]

Making the server slow for about an hour. Backing up zoklet's files (not the db) - because there are SO many files, this might take some time.

[Zok]

SSL Certs are fixed.

[Zok]

This morning we were down for a few hours. When the SSL certs were installed the file was keyed and required a pass phrase to reboot apache. We turn off apache in order to take a database dump for the daily backups. When the script turned off apache it was unable to reboot it because it needed that pass phrase. This has been fixed

[Zok]

Just made some hardware adjustments to the server. Sorry for the downtime

[Zok]

Just updated a shitton of software on the server. You probably won't notice any differences since most of the software was behind the scenes stuff. Just letting y'all know what that 10 minute downtime was.

[Zok]

Sharktech (our host) went offline for about 30 minutes just now. I'm not sure why (still waiting for details) but it looks like it's all back now

Edit: just got this from our host:

On Saturday 10/23/2010 5:10PM MST. We experienced a Edge router complete failure, upon investigation it appeared to be a UPS unit failure that caused this problem. UPS unit was replaced and systems are back up and functioning at normal level. This router was up for over 1 year, this is the first failure we have experienced on our network edge.

[Zok]

Restarted mysql and apache - the site was running pretty slow and there was no obvious reason for it. Seems to have fixed the problem for now.

[Zok]

Shortly after I restarted apache and mysql, it looks like our datacenter had network issues. No explanation from them, but they got it fixed pretty fast.

[Zok]

Just received this email:

Friday 11/12/2010 5PM MST we started noticing that our Edge router in Chicago is experiencing a problem causing the system to become unstable and spiking the CPU to %100 causing the BGP routing sessions to go unresponsive. After further investigation we have noticed a certain pattern of traffic causing this problem.

We are currently working with Cisco to have this issue resolved. We should have it resolved tonight.

[Zok]

Just received this from our datacenter:

For the next 3 days we will be doing some maintenance in our Chicago Data Center. This will be effecting all systems with IPs in the ranges:

* 204.188.x.x
* 64.32.x.x
* 208.98.x.x
* 70.39.64.0 to 70.39.95.255
* 67.21.x.x

This maintenance is intended to clean up our shelves and set the wires in a way easy to maintain and manage. The downtime any server would experience is no more than 5-10 minutes.

We sincerely apologize for this inconviniance and rest assured we are doing our best to make sure this is as seemless as possible, but we have to do this clean up at least once a year.

This includes us. Just so everyone knows!

[Zok]

Originally Posted by Zok

Just received this from our datacenter:



This includes us. Just so everyone knows!

As you can tell, we were offline for a few more than 10-15 minutes. Apparently they were reorganizing some things at the datacenter and they didn't look to see that our server was hooked back up properly. No data was lost - just offline for a while.

[Zok]

So as I'm sure some of you noticed, zoklet had been going offline for a few hours at a time over the last few days. The problem was that another machine at the datacenter was trying to allocate our main IP and was causing an IP conflict.

The issue has been resolved, so there should be no more downtime related to this.

[Zok]

Today's downtime:

There was a power interruption at the Chicago Facility. Our techs are working to bring your servers back online.

[Zok]

Tomorrow's scheduled downtime:

On 04/24/2011 11PM MST we will be performing upgrades on our network infrastructure in Chicago. The upgrades will interrupt service in our Chicago DC for 2-3 hours from the startup time. We will be performing the following upgrades to our Chicago Infrastructure:

* Upgrade Core router line card for switching uplinks.
* Upgrade switching fabric uplinks.
* Routine Maintenance on current Firewalls
* Phase II of the new firewalls implementation

[Zok]

Update from the last few days (from our host)

A) On April 23rd 8:30AM MST Chicago DC: a power surge (we are still investigating the cause) occured and at the same time triggered a battery failure on our UPS responsible for our network infrastructure.

B) On April 24th 2:10AM MST Denver DC: one of our Gig switches failed and was not responding back. Systems were moved to a different Gig switch.

C) On April 25th 12:30AM MST Chicago DC: Scheduled maintenance was initiated and completed successfully at 2:30AM MST. This maintenance included:

1. Upgrade of our core routing equipment for higher bandwidth performance
2. Organize and start phase II of adding our additional firewall farm in Chicago.
3. Upgrade our network infrastructure cross-connect

D) We are currently looking around for new providers to help us meet the demand of bandwidth from the Asian market.

[Zok]

Sorry for that brief downtime just there - I just wanted to take a snapshot of the database for development purposes. Fear not, zoklet lives on.

[Zok]

Scheduled downtime for this evening (May 4, 2012). Things should be going offline around 9pm CST and back online by midnight. The datacenter is reorganizing their equipment to prepare for a network expansion.

[Zok]

Something at our host happened (still not sure what) which caused the server to reboot. When it rebooted, it cut off some updates to the database table that holds posts and corrupted it as a result. After the server restarted, I repaired the table and got it working again. Any of the posts that were added just as the outage occurred (like, within a second or two) were probably lost.

Things seem to be running smoothly now.