Here's the display for a stat for the past 24 hours (in Graphite Composer):
Here's the display for a stat for the "past 14 days":
Not much difference there. I cannot convince Graphite to display any data for any period past the past 24 hours.
Here are the relavent entries from storage-schemas.conf (I'm using StatsD):
[stats]
pattern = ^stats.*
retentions = 10:2160,60:10080,600:262974
[stats_counts]
pattern = ^stats_counts.*
retentions = 10:2160,60:10080,600:262974
and my storage-aggregation.conf:
[min]
pattern = \.min$
xFilesFactor = 0
aggregationMethod = min
[max]
pattern = \.max$
xFilesFactor = 0
aggregationMethod = max
[sum]
pattern = \.count$
xFilesFactor = 0
aggregationMethod = sum
[default_average]
pattern = .*
xFilesFactor = 0
aggregationMethod = average
I have five or so days of data captured so far. What am I missing?
EDITED to add:
I guess I should mention that I started out with the default storage-schemas.conf and only yesterday rebuilt my whisper database files to match the above configuration. I don't think this should be relevant, but there it is.
UPDATED:
I'm using 0.9.10 of graphite-web and whisper, from PyPI, released in May 2012.
Well, this is what I get for not pasting the entire configuration. Here's what it actually looked like:
[carbon]
pattern = ^carbon\.
retentions = 60:90d
[default_1min_for_1day]
pattern = .*
retentions = 60s:1d
[stats]
pattern = ^stats.*
retentions = 10:2160,60:10080,600:262974
[stats_counts]
pattern = ^stats_counts.*
retentions = 10:2160,60:10080,600:262974
Of course, the [default_1min_for_1day] section was matching first, ahead of the other two, and so I was only getting data for the past 24 hours. Moving the catch-all to the end of the file seems to have addressed the issue.
Related
I am in the process of learning Julia and I'd like to do some buffer manipulation.
What I want to achieve is the following:
I've got a buffer that I can write to and read from at the same time, meaning that the speed with which I add a value to the Fifo buffer approximately equals the speed with which I read from the buffer. Reading and writing will happen in separate threads so it can occur simultaneously.
Additionally, I want to be able to control the values that I write into the buffer based on user input. For now, this is just a simple console prompt asking for a number, which I then want to write into the stream continously. The prompt refreshes and asks for a new number to write into the stream, but the prompt is non-blocking, meaning that in the background, the old number is written to the buffer until I enter a new number, which is then written to the buffer continuously.
This is my preliminary code for simulatenous reading and writing of the stream:
using Causal
CreateBuffer(size...) = Buffer{Fifo}(Float32, size...)
function writetobuffer(buf::Buffer, n::Float32)
while !isfull(buf)
write!(buf, fill(n, 2, 1))
end
end
function readfrombuffer(buf::Buffer)
while true
while !isempty(buf)
#show read(buf)
end
end
end
n_channels = 2
sampling_rate = 8192
duration = 2
n_frames = sampling_rate * duration
sbuffer = CreateBuffer(n_channels, n_frames)
print("Please enter a number: ")
n = parse(Float32, readline())
s1 = Threads.#spawn writetobuffer(sbuffer, n)
s2 = Threads.#spawn readfrombuffer(sbuffer)
s1 = fetch(s1)
s2 = fetch(s2)
I am not sure how to integrate the user input in a way that it keeps writing and reading the latest number the user put in. I looked at the documentation for channels, but didn't manage to get it working in a way that was non-blocking for the stream writing. I don't know that the correct approach is (channels, events, julia's multithreading) to enable this functionality.
How would I go on about to include this?
I managed to get it working, but I think it could be improved:
using Causal
CreateBuffer(size...) = Buffer{Fifo}(Float32, size...)
function writeToBuffer(buf::Buffer, n::Float32)
write!(buf, fill(n, 2, 1))
end
function readFromBuffer()
global soundbuffer
println("Starting")
sleep(0.5)
while true
while !isempty(soundbuffer)
read(soundbuffer)
end
end
println("Exiting...")
end
function askForInput()::Float32
print("Please enter a number: ")
a = parse(Float32, readline())
return(a)
end
function inputAndWrite()
global soundbuffer
old_num::Float32 = 440
new_num::Float32 = 440
while true
#async new_num = askForInput()
while (new_num == old_num)
writeToBuffer(soundbuffer, new_num)
end
old_num = new_num
println("Next iteration with number " * string(new_num))
end
end
n_channels = 2
sampling_rate = 8192
duration = 2
n_frames = sampling_rate * duration
soundbuffer = CreateBuffer(n_channels, n_frames)
s1 = Threads.#spawn inputAndWrite()
s2 = Threads.#spawn readFromBuffer()
s1 = fetch(s1)
s2 = fetch(s2)
I want to deploy influxDB with collectd.
For previous version ie 1.8 of influxDB it was supported.
Do we have support for collectd in the latest version of influxdb i.e 2.0?
This functionality was moved to telegraf. in /etc/telegraf/telegraf.conf you have to define a collectd input and an influxdb output, then it works. I haven't found an example so I had to piece it together from different sources of documentation:
[agent]
interval = "10s"
round_interval = true
metric_batch_size = 1000
metric_buffer_limit = 10000
collection_jitter = "0s"
flush_interval = "10s"
flush_jitter = "0s"
precision = ""
hostname = ""
omit_hostname = false
[[inputs.socket_listener]]
service_address = "udp://:25826"
data_format = "collectd"
collectd_auth_file = "/etc/collectd/collectd.auth"
collectd_security_level = "encrypt"
collectd_typesdb = ["/usr/share/collectd_types.db"]
collectd_parse_multivalue = "split"
[[outputs.influxdb_v2]]
urls = ["http://influxdb:8086"]
token = "$DOCKER_INFLUXDB_INIT_ADMIN_TOKEN"
organization = "$DOCKER_INFLUXDB_INIT_ORG"
bucket = "$DOCKER_INFLUXDB_INIT_BUCKET"
Is it possible to run the regex preprocessor over multiple measurements like that?
[[processors.regex]]
namepass = ["measure1", "measure2"]
[[processors.regex.fields]]
key = "agent"
pattern = '^.*$'
replacement = "NORMAL"
result_key = "agent_type"
In my case two measurements both have an Access-Log as source ([[inputs.tail]]) but I want to keep them seperate as I want to compare both eventually.
To answer my own question: I'm not sure if this is how it's meant to be but a quickfix would looke like that:
[[processors.regex]]
namepass = ["measure1"]
[[processors.regex.fields]]
key = "agent"
pattern = '^.*$'
replacement = "NORMAL"
result_key = "agent_type"
[[processors.regex]]
namepass = ["measure2"]
[[processors.regex.fields]]
key = "agent"
pattern = '^.*$'
replacement = "NORMAL"
result_key = "agent_type"
Unfortunately it contains duplicated code which is bad.
I've wrote the following code:
require 'nn'
require 'cunn'
file = torch.DiskFile('train200.data', 'r')
size = file:readInt()
inputSize = file:readInt()
outputSize = file:readInt()
dataset = {}
function dataset:size() return size end;
for i=1,dataset:size() do
local input = torch.Tensor(inputSize)
for j=1,inputSize do
input[j] = file:readFloat()
end
local output = torch.Tensor(outputSize)
for j=1,outputSize do
output[j] = file:readFloat()
end
dataset[i] = {input:cuda(), output:cuda()}
end
net = nn.Sequential()
hiddenSize = inputSize * 2
net:add(nn.Linear(inputSize, hiddenSize))
net:add(nn.Tanh())
net:add(nn.Linear(hiddenSize, hiddenSize))
net:add(nn.Tanh())
net:add(nn.Linear(hiddenSize, outputSize))
criterion = nn.MSECriterion()
net = net:cuda()
criterion = criterion:cuda()
trainer = nn.StochasticGradient(net, criterion)
trainer.learningRate = 0.02
trainer.maxIteration = 100
trainer:train(dataset)
And it must works good (At least I think so), and it works correct when inputSize = 20. But when inputSize = 200 current error always is nan. At first I've thought that file reading part is incorrect. I've recheck it some times but it is working great. Also I found that sometimes too small or too big learning rate may affect on it. I've tried learning rate from 0.00001 up to 0.8, but still the same result. What I'm doing wrong?
Thanks,
Igor
I need to supervise an real time application. This application receives 60 connections per seconds and for each I use 53 metrics.
So my simulation client sent 3180 metrics personds.
I need the lower, upper, average, median and the count_ps values. Thats why I use the "timing" type.
When I look the count_ps at the end of statsd for one metrics, i have only 40 values and not 60.
I dont find information on statsd's capacity. Maybe I overload it ^^
So could you help me, what are my options ?
I can't reduce the nomber of metrics but i don't need all informations provided by the "timing" type. Can I limit the "timing" ?
Thank you !
my configuration :
1) cat storage-schemas.conf
# Schema definitions for Whisper files. Entries are scanned in order,
# and first match wins. This file is scanned for changes every 60 seconds.
#
# [name]
# pattern = regex
# retentions = timePerPoint:timeToStore, timePerPoint:timeToStore, ...
# Carbon's internal metrics. This entry should match what is specified in
# CARBON_METRIC_PREFIX and CARBON_METRIC_INTERVAL settings
[carbon]
pattern = ^carbon\.
retentions = 60:90d
[stats]
pattern = ^application.*
retentions = 60s:7d
2) cat dConfig.js
{
graphitePort: 2003
, graphiteHost: "127.0.0.1"
, port: 8125
, backends: [ "./backends/graphite", "./backends/console" ]
, flushInterval: 60000
, debug: true
, graphite: { legacyNamespace: false, globalPrefix: "", prefixGauge: "", prefixCounter: "", prefixTimer: "", prefixSet: ""}
}
3) cat storage-aggregation.conf
# Aggregation methods for whisper files. Entries are scanned in order,
# and first match wins. This file is scanned for changes every 60 seconds
#
# [name]
# pattern = <regex>
# xFilesFactor = <float between 0 and 1>
# aggregationMethod = <average|sum|last|max|min>
#
# name: Arbitrary unique name for the rule
# pattern: Regex pattern to match against the metric name
# xFilesFactor: Ratio of valid data points required for aggregation to the next retention to occur
# aggregationMethod: function to apply to data points for aggregation
#
[min]
pattern = \.lower$
xFilesFactor = 0.1
aggregationMethod = min
[max]
pattern = \.upper$
xFilesFactor = 0.1
aggregationMethod = max
[sum]
pattern = \.sum$
xFilesFactor = 0
aggregationMethod = sum
[count]
pattern = \.count$
xFilesFactor = 0
aggregationMethod = sum
[count_legacy]
pattern = ^stats_counts.*
xFilesFactor = 0
aggregationMethod = sum
[default_average]
pattern = .*
xFilesFactor = 0.3
4) Client :
#!/usr/bin/env python
import time
import random
import statsd
import math
c = statsd.StatsClient('localhost',8125)
k = 0
nbData = 60
pause = 1
while True :
print k
k += pause
tps1 = time.clock()
for j in range (nbData):
digit = j%10 + k*10 + math.sin(j/500)
c.timing('TPS.global', digit)
c.timing('TPS.interne', digit)
c.timing('TPS.externe', digit)
for i in range(5):
c.timing('TPS.a.'+str(i), digit)
c.timing('TPS.b.'+str(i), digit)
c.timing('TPS.c.'+str(i), digit)
c.timing('TPS.d.'+str(i), digit)
c.timing('TPS.e.'+str(i), digit)
c.timing('CR.a.'+str(i), digit)
c.timing('CR.b.'+str(i), digit)
c.timing('CR.c.'+str(i), digit)
c.timing('CR.d.'+str(i), digit)
c.timing('CR.e.'+str(i), digit)
tps2 = time.clock()
print 'temps = ' + str(tps2 - tps1)
if k >= 60:
k = 0
if pause-tps2 + tps1 < 1:
time.sleep(pause-tps2 + tps1)
Edit : add client code
Without more context it's hard to say, what could be going on. Do you use sampling when sending data to StatsD? What hardware are you running StatsD on? Was your simulation all on localhost? Did you run it on a lossy connection?
At the moment there is no way of limiting timing metrics to only certain types.
Sorry to not be of more immediate help. If your problems persist, consider dropping into #statsd on Freenode IRC and ask there.
What is your CARBON_METRIC_INTERVAL set to? I suspect it needs to match the StatsD flushInterval.