Topic: Plugin Development / Plugin not working - warnings?

I’ve got a plugin that tests okay (scout -p plugin.rb) but I’m getting a warning. (I’m monkeypatching Time)

Would this cause the plugin to fail in real-world use?

Topic: Scout Client Help / undefined method `[]'

I ran into this error while setting up our Scout Client for Skribit:

F, [2008-06-28 09:00:51 #30380] FATAL -- : An HTTP error occurred:  undefined method `[]' for false:FalseClass

If you run into this error, try removing your /home/mongrel/.scout/client_history.yaml file. That worked for me.

Perhaps the client should handle the scenario where a blank client_history.yaml file is created? Not sure how it got that way.

—Calvin

Topic: Announcements / New Scout Gem Released

Scout 2.0 has been released!

See our blog post for more details:

http://blog.scoutapp.com/articles/2008/06/09/ev…

Enjoy!

Topic: System Announcements / Interruption of Service 06/11/2008

Scout experienced problems collecting data and reporting from 2:30 – 9:30 PM EST (6:30 – 13:30 GMT). During this time, intermittent data was collected from Scout clients and the Scout web interface was not available.

The problem stems from an upgrade we are still working on to provide better resources for the virtualized Scout instances across the data centers. More specifically, we’re experiencing issues running virtualized 32-bit instances on 64-bit hosts. We are now running on much better hardware, and will continue to work towards improving the reliability and speed of the Scout service.

We care deeply about reliability and rely on Scout ourselves to monitor our own servers. While we do not have a Service Level Agreement (SLA), we want to compensate you if were negatively affected by the outage. Please contact us at support@highgroove.com and we’ll take care of it.

Topic: System Announcements / Interruption of Service 5/21/2008

We had a quick outage of one of the servers in the load balance pool for a few minutes today (around 3:10pm EST / 12:10pm PST lasting for about 10-15 minutes). Technically, the one server did not go down, but rather, became so overloaded that requests timed out. That means a portion of the requests, including reports, alerts, and errors from the client were sent to a server that was unable to respond. We apologize for any inconveniences.

If your client continues to run instead of closing down, due to a hung HTTP connection, please consider upgrading the scout client using:

sudo gem install scout

to get the latest version.

As always, we care deeply about reliability and rely on Scout ourselves to monitor our own servers. While we do not have a Service Level Agreement (SLA), we want to compensate you if were negatively affected by the outage. Please contact us at support@highgroove.com and we’ll take care of it.

Topic: Plugin Development / Uptime plugin

Rafi,

Sorry for the delay – currently the uptime plugin doesn’t generate any alerts, so you’ll only get an email notification on an error.

Topic: Plugin Development / Uptime plugin

Just checking to make sure the uptime plugin does what I think, b/c I’m not sure I totally understand the email notification thresholds.

If I set up email notification for the uptime plugin (which isn’t parameterized), when will I get emailed by Scout?
Only after a reboot?
All day until uptime is >1?
Something else?

Topic: System Announcements / Interruption of Service 5/11/2008

The issue has been resolved. Clients were unable to checkin, and data was unable to be collected for about one hour.

We care deeply about reliability and rely on Scout ourselves to monitor our own servers. While we do not have a Service Level Agreement (SLA), we want to compensate you if were negatively affected by the outage. Please contact us at support@highgroove.com and we’ll take care of it.

Topic: System Announcements / Interruption of Service 5/11/2008

On Sunday May 11 2008 at around 8:00 am EST (5:00 am PST), we had a minor issue with one of the mysql db masters. We are working to resolve the issue.

Topic: Scout Web Interface Help / Batch plugin updates

Will,

Currently, you have to make the change on each plugin – here’s an idea though:

What if in the plugin settings, we add a checkbox that says “Update All Plugin Installations”. When checked, it will update all plugins that match this plugin’s URL.

Would that meet your needs?

Topic: Plugin Development / Plugin security

A couple of ideas:

A. Have the plugin read the password from a file off the disk. For each machine Scout is monitoring, put that password file somewhere only the Scout client user has access to.

B. Set the DB password in an environment variable in the Scout user’s crontab, or just in the crontab line for Scout. Use the password from env in your plugin script.

Topic: Plugin Development / Plugin security

Of course, I found the “How does Scout approach security?” mere minutes after posting this. :-) I think that answers most of my questions. Still not quote confident that I want DB passwords in my plugins. Anyone else have a favorite solution to that issue?

Topic: Plugin Development / Plugin security

What can the admins tell us about plugin security? For example, I’d like to execute some DB queries directly in a plugin (not necessary to load my whole Rails environment). But to do that, I’d need a DB password in the plugin.

I haven’t sniffed to see, but are the connections that the scout client makes encrypted?

How about the security of the plugin store at scoutapp.com? Would it be inadvisable to store proprietary information in a plugin to be distributed by the scout server?

Topic: Scout Web Interface Help / Batch plugin updates

Is there a way in the web interface (or planned support for) batch updating plugin settings?

For example, I have 13 clients set up, monitoring disk space. I’d like to change all my capacity alerts to come at 80% instead of the default 90%, or I want a plugin running on a bunch of machines to run every 5 minutes instead of the default 10 minutes.

Ideally, my account would have plugin configs for each plugin, and I could assign those configs to machines, in batch.

Do I currently need to go through each machine’s plugins and make the change individually?

Topic: System Announcements / Interruption of Service 4/28/2008

Just a follow up—we now have a solution in place to ensure that if the server hosting the community plugins becomes unavailable, we can keep on chugging!

Topic: Scout Client Help / cron interval

Randy, that’s an excellent suggestion. It will work now, but we may in the future decide not to accept report data on the time interval. Thanks!

Topic: Scout Client Help / cron interval

I am on the free plan but have cron set up to run every 10 minutes. It seems to be working, what happens if we leave it this way? It would be nice if we could run it more frequently to take advantage of scripts that monitor and restart services, but have it only accept reports at the time intervals allowed.

Topic: Scout Client Help / Scout Performance Impact

Also, if you’re worried about CPU cycles being stolen by Scout, you can always run scout using nice to give it a low priority:

/usr/bin/nice -n 10 /usr/bin/scout

We’ve found the performance impact to be very very low, nonetheless.

Topic: System Announcements / Interruption of Service 4/28/2008

Just to clarify—only one instance of the scout virtualized setup experienced problems and had to be rebooted. Only community plugins were affected by the downtime, which happens to be a majority of the plugins, and thus a majority of clients did not run plugins during this time. Clients were still checking in, thus monitoring still functioned. We are currently working on a solution to make sure this does not happen again.

Topic: System Announcements / Interruption of Service 4/28/2008

Scout was inaccessible from 1:50 – 2:28 PM PST (20:50 – 21:28 GMT). During this time, no data was collected from Scout clients and the Scout web interface was not available.

We care deeply about reliability and rely on Scout ourselves to monitor our own servers. While we do not have a Service Level Agreement (SLA), we want to compensate you if were negatively affected by the outage. Please contact us at support@highgroove.com and we’ll take care of it.

We have a load-balanced setup that is designed to handle a failure like this. Obviously, it did not perform. We will update this thread when we determine the cause.

Topic: Scout Client Help / Scout Performance Impact

Christian,

We built Scout so that it’s impact on performance would be very small. Here’s the basics:

• In our tests, the Scout client used btw 5-15 MB of memory when running
• The Scout client only runs when it’s executing the plugins (ex: every 30 minutes on the free plan) so it doesn’t use resources when it’s not active.
• The impact on the server load wasn’t detectable

The client is extremely light – most of the work is done by the plugins. You could build a plugin that is extremely intensive – one that requires a lot of libraries maybe – and it will certainly use more resources. So in the end, it’s up to you.

Also note that you can use Scout to monitor itself – you can use the server load plugin and monitor the memory usage of the “ruby” process to get a feel.

Topic: Scout Client Help / Scout Performance Impact

I have a basic question about scout that I didn’t find in the help section.

Given a web app on a server that doesn’t use Ruby, what the performance impact of installing not just the Scout Client, but also the Ruby engine as well? Can you give us some numbers so we have a sense of it?

Topic: Plugin Development / Plugins WANTED!

Matt—

I haven’t checked out m|monit, but I actually do use monit to keep all the components running for Scout. Unfortunately though, getting an e-mail about a process ballooning in memory and then restarting doesn’t really help me diagnose why it did it in the first place, or what was going on at the time. I’m with you (and probably preaching to the choir about how Scout helps to alleviate this problem by showing you what other factors could have contributed to this), though!

And having a scout plugin that works in conjunction with monit is a fantastic idea. Brilliant, even.

I am testing a new plugin called, simply enough: Keep Process Running and it couldn’t be simpler. I think interrogating the monit file is an awesome idea too.

Keep ‘em coming! Thanks for the feedback!

Topic: Plugin Development / Plugins WANTED!

What about an interface to monit? The monit folk recently released their “m|monit” project (http://tildeslash.com/mmonit/).

Scout looks a heck of a lot cooler, has a great, defined plug-in interface, is much further along, and doesn’t require another process running (should monit monitor m|monit, too??)—in short, it seems like a far superior solution.

What about a plugin that lets you interface with the monit process already running on our servers? I see the plugin library already has a simple script to restart mongrel. But the monit process monitoring/restarting syntax is significantly more advanced: why re-invent the wheel?

It could start out simply enough: interrogate the monit conf file and execute commands held within, but seems a short hop to then replace all that m|monit does.

Interested in the everyone’s thoughts!!

Thanks,

—
Matt

Topic: Graphing / Graphs not in my timezone

Rafi,

Yes – we have a number of updates to graphing that we’ll be implementing late next week. This is one of the pieces I’ll be fixing (I’ll put the graphs in your time zone).