Is there any way to filter metrics in Graphite while ignoring the hierarchy?
For example:
Say I have the following metrics:
stats_count.A.B.TestMetric
stats_count.A.TestMetric
stats.A.B.TestMetric
stats.A.B.TestMetric
How can I sum TestMetric under stats_count only?
I tried the followings with no success:
stats_counts.*.*.TestMetric - obviously this won't work...
sumSeriesWithWildcards(stats_counts.[A-Z\.]*[A-Z]*.TestMetric,1)
sumSeriesWithWildcards(stats_counts.[A-Z]*[.]*[A-Z]*.TestMetric,1)
Any ideas? Is it possible at all?
I have a graphite installation (version 0.9.9) where I create metrics on a lot of small systems
For example, I have 2 installations of a postgresql database (postgresql-1 and postgresql-2) where the second is a slave replication of the first. The first database is used for day to day use while the second is a hot standby used mostly by reporting systems and debugging queries
I think the following example is somewhat what you want to do. The image is the amount of connections on both databases. The blue line is on the first, the green line is on the second while the red line is the sum of both series, giving the total amount of connections
database.postgresql-1.connection.ALL.value # blue line
database.postgresql-2.connection.ALL.value # green line
sumSeries(database.postgresql-*.connection.ALL.value) # red line
Your problem is that your series have different path levels (I tried to avoid that on my series names as indeed it causes problems). I dont see any other option than writing something like this:
given
stats_count.A.B.TestMetric
stats_count.A.TestMetric
stats.A.B.TestMetric
stats.A.B.TestMetric
sum stats_count.**.TestMetric metrics using
sumSeries(stats_count.*.TestMetric, stats_count.*.*.TestMetric)
Graphite still needs a lot of improvement and unfortunately the development is going quite slowly (for instance version 0.9.10 is the latest release, has problems to install and is from 1 year ago). I am indeed considering forking/contributing to this project
Related
I am trying to get a more responsive idea of the system run queue length to see if load balancing based on the one minute load average from sysinfo() is having issues caused by the client processes perhaps looking in lockstep...
I've managed to find /proc/schedstats, and it looks to be what I'm looking for, but...
I want to make sure I base my values on the actual interval between polls of /proc/schedstat, instead of potential processing overhead (it's a shell script).
Now for the question: What is the unit of measurement used for the "timestamp" value at the top of the /proc/schedstats file? It's sure not nanoseconds, because the value is somewhere between 258 and 260 when my script loops through with a sleep 1 between loops.
Inspecting kernel sources sched/stats.c shows that the timestamp field is in jiffies.
What is the unit of measurement used for the "timestamp" value at the top of the /proc/schedstats file?
It's 1/HZ second, typically with HZ=300, the unit would be 3.333 miliseconds if I'm counting right.
I've gone through their documentation (here & here), but I don't see a simple way to tweak this. For instance if I wanted the lines to show time spent in minutes and the bubbles to show the distinct # of cases that appeared there instead of absolute/relative frequency - ex: when looking at incident case data.
Do I have to render the process map myself via DiagrammeR? I was trying to find some a similar example in DiagrammeR but I couldn't find much.
Any tips / good examples I could reference? I suppose its because this is quite new. I did find this older article but I wasn't sure how to connect the dots. Is it worth continuing my analysis in bupaR or should I leverage a different process map generating library?
This wasn't possible so far, but it is now due to a recent change on the github repo. You can now have frequencies on nodes and times on arrow (or vice versa).
What you do is, first install the github version of the package:
devtools::install_github("gertjanssenswillen/processmapr")
then, use the new type_edges and type_nodes arguments, in the same way as you would use the type argument before.
data %>% process_map(type_nodes = frequency(value = "absolute_case"), type_edges = performance(FUN = mean, units = "min"))
I have 2 moderate-size datasets that I am using in R. I want to check one dataset if its referenece number matches with the reference numbers in the other dataset and if so, allot a column in the second dataset which contains the value present in the column in the other dataset.
ghi2$state=ifelse(b1$accntnumber %in% ghi2$referencenumber,b1$address,0)
Every time I am running this code, my RStudio hangs up and is unresponsive for a long time. Is it because its taking the time to process the command or is my command wrong.
I am using a 2GB RAM system so I think R hangs up. Should I use the == operator instead of %in%? Would I get the same result?
1. Should I use the == operator instead of %in%?
No (!). See #2.
2. Would I get the same result?
No. The order and position have to match with ==. Also, see #Akrun's comment.
3. How to make it faster and/or deal with RStudio freezing
If RStudio freezes you can save your log file info, send it to the RStudio team who will quickly respond, and also you could bring your log files here for help.
Beyond that, general Big Data rules apply. Here are some tips:
Try data.table
Try it on the command line instead of RStudio
Watch your Resource Monitor (or whatever you use to monitor resources) and observe the memory and CPU usage
If it's a RAM issue you can
a. use a cloud account to get more RAM
b. buy some more RAM (just sayin')
c. use 64-bit R and increase the RAM available to R to its max if it's not already
If it's a CPU issue you can consider parallelization
If any of these ID's are being repeated (and this makes sense in the context of your specific use-case) you can use unique to avoid redundant comparisons
There are lots of other tips you can find in pre-existing Big Data Q&A's on SO as well.
Solution: As suggested by user Andy in the comments, an update to the newest version of Octave (at the moment: octave-4.0.1-rc4) fixed the problem and the plot could be saved as PNG.
I have a large-ish amount of data that I plot in Octave. But when I try to save the image, the program crashes without any explanation or real error message. My Octave is version 4.0 and it's running on Win 8.1, the graphics_toolkit is qt.
Saving smaller amounts of data has worked so far, but somehow I seem to have reached a size where the plot can be drawn but not saved.
First, I load the data from several files listed in the vector inputs:
data = [];
for i = 1:length(inputs)
data = [data; load(inputs{i})];
endfor
The result is a 955.524 x 7 matrix containing numbers. Loading alone takes a while on my system (several minutes), but eventually succeeds. I then proceed to plot the data:
hold on;
for j = 1:length(data(1,:))
curenntColumn = normalize(data(:,j)); % make sure all data is in the same range
plot(1:length(curenntColumn), curenntColumn, colours{j}); % plot column with distinct colour
endfor
hold off;
This results in a plot being drawn as Figure 1 that shows all 955.524 entries of each of the seven columns correctly in a distinct colour. If the program ends here, it exits properly. However, if I add
print("data.png");
Octave will keep running after opening the plot window and eventually crash with a simple "program does not work anymore" error message. The same happens if I try to save manually from the File->Save menu (which offers saving as PDF). Even just touching and moving the plot window takes a few seconds.
I tried using gnuplot and fltk as graphics_toolkit, but the latter does not even open a plot window, and the former seems to be broken (crashes on the attempt of plotting even simple data like plot(1:10,1:10);).
Now, I could screenshot the plot and try to work with that, but I'd really rather have it be saved automatically. Also, I find it weird that displaying the curves is possible, but not saving said display. As it works for smaller amounts of data, maybe I just need to somehow allocate more resources to Octave?
It (4.2.2 version) crashes with my Linux Mint. just a simple graph, and it crashed two times in a row. I am going back to R. I had my hopes up as I wanted to review the Numerical Analysis Using Matlab text.
Wait, come to think of it, the Studio Version of R crashes when I try to use it but not when I run the same program from the command line, so I will go back (one more time) and try to run a plot totally from the command line. The Linux Mint requires a 2 CPU 64 bit, and I just have the 64 bit single CPU.
The following data comes from a mobile phone provider, it's a list of kb's downloaded at a certain time, usually on a per minute basis.
It's not the average, not the max, but the total of that time interval, which allows to track the data consumption precisely. These graphs were made with PIL, and instead of showing spikes to indicate a large data consumption, large steps can be seen, which is much more revealing, because it doesn't just tell "much happened here", but "exactly this much happened here". For example second graph Sat 10 at night 100mb. A rate-change graph wouldn't be as informative.
I'm also trying to find a way to do this with rrd.
I was mislead when using the COUNTER to track my networks data usage into thinking that I would be able to precisely compute the monthly/weekly accumulated data usage, but now it turned out to be a wrong assumption.
How I store my data in rrd in order to be able to easily generate graphs like below? Would that be by using ABSOLUTE and before updating it I would subtract the previous insertion value? Would that be precise down to the byte when checking the monthly usage?
You can add up all the value in your chart quite easily:
CDEF:sum=data,$step_width,*,PREV,ADDNAN
if your chart covers just one month, that should be all you have todo. If you want to have it cover multiple months, you will have to use a combination of IF and TIME operators to reset the line to 0 at the start of the month.
Version 1.5.4 will contain an additional operator called STEPWIDTH, which pushes the step width onto the stack, making this even simpler.
This is a common question which very few answers online but I first encountered a method to do this with RRD in 2009.
The DS type to use is a GAUGE and in your update script manually handle resetting the GAUGE to 0 at the start of the month for monthly usage graphs.
Then came along RRDTool's ' mrtg-traffic-sum ' package.
More recently I've had to monitor both traffic bandwidth and traffic volume so I created a standard RRD for that first and confirmed that was working.
So with the bandwidth being sampled (captured to the RRD), then use the mrtg-traffic-sum tool to generate the stat's needed as in the example below then pump them into another RRD created with just the GAUGE DS type and just LAST (no need for MIN/AVG/MAX).
This allows using RRDs to collect both traffic bandwidth as well as monthly traffic volumes / traffic quota limits.
root#server:~# /usr/bin/mrtg-traffic-sum --range=current --units=MB /etc/mrtg/R4.cfg
Subject: Traffic total for '/etc/mrtg/R4.cfg' (1.9) 2022/02
Start: Tue Feb 1 01:00:00 2022
End: Tue Mar 1 00:59:59 2022
Interface In+Out in MB
------------------------------------------------------------------------------
eth0 0
eth1 14026
eth2 5441
eth3 0
eth4 15374
switch0.5 12024
switch0.19 151
switch0.49 1
switch0.51 0
switch0.92 2116
root#server:~#
From mrtg-traffic-sum just write up a script that will populate your 2nd rrd with these values & presto you have a traffic volume / quota graph also.