Interruption of Service 06/11/2008 replied by seebq @ Wed, 11 Jun 2008 16:01:00 -0000

seebq — Wed, 11 Jun 2008 16:01:00 -0000

Scout experienced problems collecting data and reporting from 2:30 – 9:30 PM EST (6:30 – 13:30 GMT). During this time, intermittent data was collected from Scout clients and the Scout web interface was not available.

The problem stems from an upgrade we are still working on to provide better resources for the virtualized Scout instances across the data centers. More specifically, we’re experiencing issues running virtualized 32-bit instances on 64-bit hosts. We are now running on much better hardware, and will continue to work towards improving the reliability and speed of the Scout service.

We care deeply about reliability and rely on Scout ourselves to monitor our own servers. While we do not have a Service Level Agreement (SLA), we want to compensate you if were negatively affected by the outage. Please contact us at support@highgroove.com and we’ll take care of it.

Interruption of Service 5/21/2008 replied by seebq @ Wed, 21 May 2008 19:26:33 -0000

seebq — Wed, 21 May 2008 19:26:33 -0000

We had a quick outage of one of the servers in the load balance pool for a few minutes today (around 3:10pm EST / 12:10pm PST lasting for about 10-15 minutes). Technically, the one server did not go down, but rather, became so overloaded that requests timed out. That means a portion of the requests, including reports, alerts, and errors from the client were sent to a server that was unable to respond. We apologize for any inconveniences.

If your client continues to run instead of closing down, due to a hung HTTP connection, please consider upgrading the scout client using:

sudo gem install scout

to get the latest version.

As always, we care deeply about reliability and rely on Scout ourselves to monitor our own servers. While we do not have a Service Level Agreement (SLA), we want to compensate you if were negatively affected by the outage. Please contact us at support@highgroove.com and we’ll take care of it.

Interruption of Service 5/11/2008 replied by seebq @ Sun, 11 May 2008 13:26:20 -0000

seebq — Sun, 11 May 2008 13:26:20 -0000

The issue has been resolved. Clients were unable to checkin, and data was unable to be collected for about one hour.

Interruption of Service 5/11/2008 replied by seebq @ Sun, 11 May 2008 13:07:36 -0000

seebq — Sun, 11 May 2008 13:07:36 -0000

On Sunday May 11 2008 at around 8:00 am EST (5:00 am PST), we had a minor issue with one of the mysql db masters. We are working to resolve the issue.

Interruption of Service 4/28/2008 replied by seebq @ Fri, 02 May 2008 22:45:30 -0000

seebq — Fri, 02 May 2008 22:45:30 -0000

Just a follow up—we now have a solution in place to ensure that if the server hosting the community plugins becomes unavailable, we can keep on chugging!

Interruption of Service 4/28/2008 replied by seebq @ Sat, 26 Apr 2008 01:56:19 -0000

seebq — Sat, 26 Apr 2008 01:56:19 -0000

Just to clarify—only one instance of the scout virtualized setup experienced problems and had to be rebooted. Only community plugins were affected by the downtime, which happens to be a majority of the plugins, and thus a majority of clients did not run plugins during this time. Clients were still checking in, thus monitoring still functioned. We are currently working on a solution to make sure this does not happen again.

Interruption of Service 4/28/2008 replied by itsderek23 @ Fri, 25 Apr 2008 22:13:03 -0000

itsderek23 — Fri, 25 Apr 2008 22:13:03 -0000

Scout was inaccessible from 1:50 – 2:28 PM PST (20:50 – 21:28 GMT). During this time, no data was collected from Scout clients and the Scout web interface was not available.

We have a load-balanced setup that is designed to handle a failure like this. Obviously, it did not perform. We will update this thread when we determine the cause.