Possible to change update interval for munin graphs? - munin

I will like to munin to graph after every two minutes to increase the granularity. There are some links pointing out that its somehow possible with munin 2.0 , any idea how?

I haven't tried it, but this question in the FAQ suggests you cannot:
Munin runs at an interval of every five minutes (*/5) on debian systems by default. Is it possible to change this interval to an arbitrary value?
Just edit /etc/cron.d/munin.
However, this won't change Munin's (or rather RRD's) granularity; all RRD files are constructed to create 5 minutes averages, and no matter how often you update the RRD files the output won't be (much) different. Changing this (default) behaviour has been proposed in Ticket #5.
However, if you look at ticket #5, the last post (closing it) is:
current trunk fully supports custom update_rate, and increase resolution of .rrd files accordingly.
So, if you're using a fairly recent version of munin, it might be worth a try.

Increasing the resolution of Munin data capture is accomplished at the plugin level, not the munin core level. This means that it is not possible to simply change an entire installation to increase resolution. Instead, each plugin must be modified independently.
The two critical plugin config values that determine resolution are
graph_data_size
update_rate # in seconds
graph_data_size determines the size and structure of the RRD file used to store captured data.
update_rate indicates how frequently new data is expected.
Since cron has a lower bound of one minute as its tightest possible resolution, that's the easiest to achieve. Simply update the cron to run every minute and modify the plugin config to include these values
graph_data_size custom 1d, 1m for 1w, 5m for 1t, 15m for 1y
update_rate 60
Sub-minute resolution requires caching datapoints in between cron runs and embedding time data (seconds since epoch). I provide full details in these three posts:
http://software.danielwatrous.com/understanding-munin-plugins/
http://software.danielwatrous.com/increase-munin-resolution-to-one-minute/
http://software.danielwatrous.com/increase-munin-resolution-to-sub-minute/

Munin uses cron job to run itself every 5 minutes by default. You can change it in the crontab. On my CentOS system it is located in /etc/cron.d/munin and looks like this:
#
# cron-jobs for munin
#
MAILTO=root
*/5 * * * * /usr/bin/munin-cron
I have changed it in the past to
*/1 * * * * /usr/bin/munin-cron
However since using Amazon AWS's monitoring system, I don't need Munin that often and now keep it at 10 minutes on the servers where I use it.

zeeshan said:
Munin uses cron job to run itself every 5 minutes by default. You can
change it in the crontab. On my CentOS system it is located in
/etc/cron.d/munin and looks like this:
#
cron-jobs for munin
#
MAILTO=root
*/5 * * * * /usr/bin/munin-cron
I have changed it in the past to
*/1 * * * * /usr/bin/munin-cron
However since using Amazon AWS's monitoring system, I don't need Munin
that often and now keep it at 10 minutes on the servers where I use
it.
You are about munin-server, because munin-agent does not have cron jobs.
Your graphs will not take a step to less than 5 min.
To change graph period ( step ) look here:
http://munin-monitoring.org/wiki/faq#Q:CanImakeagraphshowvaluesperminuteinsteadofpersecond
Sample:
[server.example.com]
address 10.0.0.1
postfix_mailstats.graph_period second
to set graph period to seconds. But be sure, that you have changed /etc/cron.d/munin for minimal time interval like graph_period in /etc/munin/munin.conf

Related

Setting a new label on all nodes takes too long in a huge graph

I'm working on a graph containing about 50 million nodes and 40 million relationships.
I need to update every node.
I'm trying to set a new label to these nodes, but it's taking too long.
The label applies to all 50 million nodes, so the operation never ends.
After some research, i found out that Neo4j treats this operation as a single transaction (i don't know if optimistic or not), keeping the changes uncommitted, until the end (which will never happen in this fashion).
I'm currently using Neo4j 2.1.4, which has a feature called "USING PERIODIC COMMIT" (already present in earlier versions). Unfortunately, this feature is coupled to the "LOAD CSV" feature, and not available to every cypher command.
The cypher is quite simple:
match n set n:Person;
I decided to use a workaround, and make some sort of block update, as follows:
match n
where not n:Person
with n
limit 500000
set n:node;
It's ugly, but i couldn't come up with a better solution yet.
Here are some of my confs:
== neo4j.properties =========
neostore.nodestore.db.mapped_memory=250M
neostore.relationshipstore.db.mapped_memory=500M
neostore.propertystore.db.mapped_memory=900M
neostore.propertystore.db.strings.mapped_memory=1300M
neostore.propertystore.db.arrays.mapped_memory=1300M
keep_logical_logs=false
node_auto_indexing=true
node_keys_indexable=name_autocomplete,document
relationship_auto_indexing=true
relationship_keys_indexable=role
execution_guard_enabled=true
cache_type=weak
=============================
== neo4j-server.properties ==
org.neo4j.server.webserver.limit.executiontime=20000
org.neo4j.server.webserver.maxthreads=200
=============================
The hardware spec is:
RAM: 24GB
PROC: Intel(R) Xeon(R) X5650 # 2.67GHz, 32 cores
HDD1: 1.2TB
In this environment, each block update of 500000 nodes took from 200 to 400 seconds. I think this is because every node satisfies the query at the start, but as the updates take place, more nodes need to be scanned to find the unlabeled ones (but again, it's a hunch).
So what's the best course of action whenever an operation needs to touch every node in the graph?
Any help towards a better solution to this will be appreciated!
Thanks in advance.
The most performant way to achieve this is using the batch inserter API. You might use the following recipe:
take a look at http://localhost:7474/webadmin and note the "node count". In fact it's not the number of nodes it's more the highest node id in use - we'll need that later on.
make sure to cleanly shut down your graph database.
take a backup copy of your graph.db directory.
write a short piece of java/groovy/(whatever jvm language you prefer...) program that performs the following tasks
open your graph.db folder using the batch inserter api
in a loop from 0..<node count> (from step above) check if the node with given id exists, if so grab its current labels and amend the list by the new label and use setNodeLabels to write it back.
make sure you run shutdown with the batchinserter
start up your Neo4j instance again

How to use VSTS Loadtest Goal based load pattern to achieve a constant test per second

I am using Visual Studio TS Load Test for running WebTest (one client/controls hitting one server). How can I configure goal based load pattern to achieve a constant test / second?
I tried to use the counter 'Tests/Sec' under 'LoadTest:Test' but it does not seem to do anything.
I've recently tested against the Tests / Sec, and I've confirmed it working.
For the settings on the Goal Based Load Pattern, I used:
Category: LoadTest:Test
Counter: Tests/Sec
Instance: _Total
When the load test starts, verify it doesn't show an error re: not being able to access that Performance Counter.
Tests I ran for my own needs:
Set Initial User Load quite low (10), and gave it 5 minutes to see if
it would reach the target Tests / Sec target, and stabilise. In my case, it stabilised after about 1 minute 40.
Set the Maximum User Count [Increment|Decrement] to 50. Turns out the
user load would yo-yo up and down quite heavily, as it would keep
trying to play catch-up. (As the tests took 10-20 seconds each)
Set the Initial User Load quite close to the 'answer' from test 1,
and watched it make small but constant adjustments to the user
volume.
Note: When watching the stats, watch the value under "Last". I believe the "Average" is averaged over a relatively long period, and may appear out of step with the target.

How can I configure Munin to give me a total of all my cloud servers?

I have a dozen load balanced cloud servers all monitored by Munin.
I can track each one individually just fine. But I'm wondering if I can somehow bundle them up to see just how much collective CPU usage (for example) there is among the cloud cluster as a whole.
How can I do this?
The munin.conf file makes it easy enough to handle this for subdomains, but I'm not sure how to configure this for simple web nodes. Assume my web nodes are named, web_node_1 - web_node_10.
My conf looks something like this right now:
[web_node_1]
address 10.1.1.1
use_node_name yes
...
[web_node_10]
address 10.1.1.10
use_node_name yes
Your help is much appreciated.
You can achieve this with sum and stack.
I've just had to do the same thing, and I found this article pretty helpful.
Essentially you want to do something like the following:
[web_nodes;Aggregated]
update no
cpu_aggregate.update no
cpu_aggregate.graph_args --base 1000 -r --lower-limit 0 --upper-limit 200
cpu_aggregate.graph_category system
cpu_aggregate.graph_title Aggregated CPU usage
cpu_aggregate.graph_vlabel %
cpu_aggregate.graph_order system user nice idle
cpu_aggregate.graph_period second
cpu_aggregate.user.label user
cpu_aggregate.nice.label nice
cpu_aggregate.system.label system
cpu_aggregate.idle.label idle
cpu_aggregate.user.sum web_node_1:cpu.user web_node_2:cpu.user
cpu_aggregate.nice.sum web_node_1:cpu.nice web_node_2:cpu.nice
cpu_aggregate.system.sum web_node_1:cpu.nice web_node_2:cpu.system
cpu_aggregate.idle.sum web_node_1:cpu.nice web_node_2:cpu.idle
There are a few other things to tweak the graph to give it the same scale, min/max, etc as the main plugin, those can be copied from the "cpu" plugin file. The key thing here is the last four lines - that's where the summing of values from other graphs comes in.

Changing between images based on date

So I have a website which has three spaces for pictures: Space A, Space B, and Space C.
In Space A, I have 20 images (Images A1-A20) which I would like to rotate daily.
In Space B, I have 20 images (Images B1-B20) which I would like to change every 13 days.
In Space C, I have 20 images (Images C1-C20) which I would like to change every 20 days.
Therefore, let's say for instance I wanted my site to start based on Jan 1, 2000.
Spaces A,B,&C would have the following images based on the date:
1/1 - A1,B1,C1
1/2 - A2,B1,C1
1/3 - A3,B1,C1
...
1/12 - A12,B1,C1
1/13 - A13,B1,C1
1/14 - A14,B2,C1
1/15 - A15,B2,C1
...
1/20 - A20,B2,C1
1/21 - A1,B2,C2
1/22 - A2,B2,C2
1/23 - A3,B2,C2
...
1/26 - A6,B2,C2
1/27 - A7,B3,C2
1/28 - A8,B3,C2
etc.
Therefore, I think I need a DateTime program that finds the number of elapsed days between the starting date (Jan 1, 2000), and the date listed on the computer, divides by the appropriate number (20,260,400), uses the remainder or fraction to determine which picture is appropriate and picks it to display, but haven't a clue how to write it / where to start.
Suggestions appreciated.
Thanks, NP
If you've been looking for a reason to learn php or perl or similar, this is it. If not, then try rent-a-coder or similar. There's a lot more to getting your first program working than just the code--you need to figure out how to get the software loaded and working correctly on your server.
The good news is that you seem to know your problem statement well. I'd go to a bookstore and look at the php or perl books until you see one that seems helpful, then go for it. And we here at SO stand ready to answer all questions from true seekers...
ps. For your problem, php would be a good starting bet if you're using a Linux web server.
Larry K is correct in that you have the problem pretty well defined and will need some sort of program to do this. PHP and Perl would be good choices, but there are ways in which you could do this with virtually any language: Python, C, Ruby, .... One technique would be to run a cron job (assuming Linux or another Unix variant) to re-write your HTML file every day (at midnight or thereabouts). Another would be to write a CGI script to determine the proper images when the page is requested. The PHP way (or Perl or Python if using mod_perl or mod_python) would embed (or pull in) the code directly in the web page and execute when the page is requested. I suggest you play around and pick what works best for you and your situation.
FWIW, if it were me, and I could do it any way I liked, I'd probably write a Python script to run as a cron job. Python just because that's currently my favorite language, and doing it by a cron job minimizes the processing required to do the job by doing it just once per day. However, some web hosting companies don't provide cron (or Python :-( ); if that were the case, I'd do it with PHP embedded in the web page, assuming I could use PHP (which seems ubiquitous with web hosting companies).

Get a Unix script to run at exactly the same time every time

I am writing a script to capture disk usage on a system (yes, I know there is software that can do this). For database reporting purposes, I want the interval between data points to be as equal as possible. For example, if I am polling disk usage every 10 minutes, I want every data point to be YYYY-MM-DD HH:[0-5]0:00. If I'm am polling every 5 minutes, it would be YYYY-MM-DD HH:[0-5][05]:00.
If I have a ksh script (or even a Perl script) to capture the disk usage, how can I let the script come active and wait for the next "Poll time" before taking a snapshot, and then sleep for the correct number of seconds until the next "Poll time". If I am polling every 5 minutes, and it is 11:42:00, then I want to sleep for 180 seconds so it will take a snapshot at 11:45:00 - and then sleep for 5 minutes so it will take another snapshot at 11:50:00.
I wrote a way that works if my poll time is every 10 minutes, but if I change the poll time to a different number, it doesn't work. I would like it to be flexible on the poll time.
I prefer to do this in shell script, but if it is way too much code, Perl would be fine too.
Any ideas on how to accomplish this?
Thanks in advance!
Brian
EDIT: Wow - I left out a pretty important part - that cron is disabled, so I will not be able to use cron for this task. I am very sorry to all the people who gave that as an answer, because yes, that is the perfect way to do what I wanted, if I could use cron.
I will be using our scheduler to kick off my script right before midnight every day, and I want the script to handle running at the exact "poll times", sleeping in between, and exiting at midnight.
Again, I'm very sorry for not clarifying on crontabs.
cron will do the job.
http://en.wikipedia.org/wiki/Cron
Just configure it to run your ksh script at the times you need and you are done
You might want to consider using cron. This is exactly what it was made for.
If I were doing this, I would use the system scheduler (cron or something else) to schedule my program to run every 180 seconds.
EDIT: I might have misunderstood your request. Are you looking more for something along the following lines? (I suspect there is a bug or two here):
ANOTHER EDIT: Remove dependency on Time::Local (but now I suspect more bugs ;-)
#!/usr/bin/perl
use strict;
use warnings;
use POSIX qw( strftime );
my $mins = 5;
while ( 1 ) {
my ($this_sec, $this_min) = (localtime)[0 .. 1];
my $next_min = $mins * ( 1 + int( $this_min / $mins ) );
my $to_sleep = 60 * int( $next_min - $this_min - 1 )
+ 60 - $this_sec;
warn strftime('%Y:%m:%d %H:%M:%S - ', localtime),
"Sleeping '$to_sleep' seconds\n";
sleep $to_sleep;
}
__END__
Have it sleep for a very short time, <=1 sec, and check each time whether poll time has arrived. Incremental processor use will be negligible.
Edit: cron is fine if you know what interval you will use and don't intend to change frequently. But if you change intervals often, consider a continuously running script w/ short sleep time.
Depending on how fine grained your time resolution needs to be, there may be a need to write your script daemon style. Start it once, while(1) and do the logic inside the program (you can check every second until it's time to run again).
Perl's Time::HiRes allows very fine granularity if you need it.

Resources