I'm trying to setup Bosun and Graphite to alert on error ratio, compiled from two different sources: API traffic and web app traffic. Here's what I have now:
$web_rate = avg(graphite("sumSeries(collectd.*.statsd.web.*.rate)", "5m", "", ""))
$api_rate = avg(graphite("sumSeries(collectd.*.statsd.api.*.rate)", "5m", "", ""))
$web_error_rate = avg(graphite("sumSeries(collectd.*.statsd.web.*.errorRate)", "5m", "", ""))
$api_error_rate = avg(graphite("sumSeries(collectd.*.statsd.api.*.errorRate)", "5m", "", ""))
$total_rate = ungroup($web_rate) + ungroup($api_rate)
$total_error_rate = ungroup($web_error_rate) + ungroup(api_error_rate)
$error_ratio = $total_error_rate / $total_rate
Our counters don't exist in graphite until they are non-zero, so for our pre-production environment, the above fails with the following:
ungroup: requires exactly one group
When I look in the expression browser, the graphite(...) call is returning, as expected, an empty set but the result of avg(graphite(...)) displays nothing.
Does anyone know how to handle this?
In case there is no data in Graphite for the metric and the specified timeframe, it will return NaN, which is nothing.
If you try to ungroup NaN, you will get the following error:
ungroup: requires exactly one group
Use nv function to protect yourself from this error. Nv function will replace possible NaN with a specified value:
nv($result, 0)
Now you can safely ungroup:
ungroup(nv($result, 0))
Related
Trying to cut a bunch of audio (.WAV) files into smaller samples in R. For this example, I'm using a loop to cut out 1 minute samples at 140 minutes.
For some files, the recording ends before 140 minutes due to an error in the recording device. When this occurs, an error appears -- and the loop stops. I'm trying to make it so the loop continues by using the try or tryCatch function however keep getting errors.
The code is as follows:
for(i in 1:length(AR_CD288)){
CUT_AR288_5 <- try({readWave(AR_CD288[i], from = 140, to = 141, units = "minutes")})
FILE.OUT_AR288_5<- sub("\\.wav$", "_140.wav", AR_CD288)
OUT.PATH_AR288_5 <- file.path("New files", basename(FILE.OUT_AR288_5))
writeWave(CUT_AR288_5, extensible=FALSE, filename = OUT.PATH_AR288_5[i])
}
I get the following two errors from the code:
Error in readBin(con, int, n = N, size = bytes, signed = (bytes != 1), :
invalid 'n' argument
Error in writeWave(CUT_AR288_5, extensible = FALSE, filename = OUT.PATH_AR288_5[i]) :
'object' needs to be of class 'Wave' or 'WaveMC
The loop still saves some samples into the "New files" directory, however, once the loop reaches a file <140 minutes, the loop stops.
I am very stuck! Any help would be greatly appreciated.
Cheers.
When I use try, I always do one (or both) of:
check the return value to see if it inherits "try-error", indicating that the command failed; or
add try(., silent = TRUE), indicating that I don't care if it succeeded (but this implies that I will not use its return value, either).
Try this:
for (i in seq_along(AR_CD288)) {
CUT_AR288_5 <- try({
readWave(AR_CD288[i], from = 140, to = 141, units = "minutes")
}, silent = TRUE)
if (!inherits(CUT_AR288_5, "try_error")) {
FILE.OUT_AR288_5 <- sub("\\.wav$", "_140.wav", AR_CD288)
OUT.PATH_AR288_5 <- file.path("New files", basename(FILE.OUT_AR288_5))
writeWave(CUT_AR288_5, extensible = FALSE, filename = OUT.PATH_AR288_5[i])
}
}
Three notes:
I changed 1:length(.) to seq_along(.); the latter is more resilient in an automated use when it is feasible that the vector might be length 0. For example, if AR_CD288 can ever be length 2, intuitively we expect 1:length(AR_CD288) to return nothing so that the for loop will not run; unfortunately, it resolves to 1:0 which returns a vector of length 2, which will often fail (based on whatever code is operating in the loop). The use of seq_along(.) will always return a vector of length 0 with an empty input, which is what we need. (Alternatively and equivalent, seq_len(length(AR_CD288)), though that's really what seq_along is intended to do.)
If you do not add silent=TRUE (or explicitly add silent=FALSE), then you will get an error message indicating that the command failed. Unfortunately, the error message may not indicate which i failed, so you may be left in the dark as far as fixing or removing the errant file. You may prefer to add an else to the if (inherits(.,"try-error")) clause so that you can provide a clearer error, such as
if (inherits(CUT_AR288_5, "try_error")) {
warning("'readWave' failed on ", sQuote(AR_CD288[i]), call. = FALSE)
} else {
FILE.OUT_AR288_5 <- sub("\\.wav$", "_140.wav", AR_CD288)
# ...
}
(noting that I put the "things worked" code in the else clause here ... I find it odd to do if (!...) {} else {}, seems like a double-negation :-).
The choice to wrap one function or the whole block depends on your needs: I tend to prefer to know exactly where things fail, so the will-possibly-fail functions are often individually wrapped with try so that I can react (or log/message) accordingly. If you don't need that resolution of error-detection, then you can certainly wrap the whole code-block in a sense:
for (i in seq_along(AR_CD288)) {
ret <- try({
CUT_AR288_5 <- readWave(AR_CD288[i], from = 140, to = 141, units = "minutes")
FILE.OUT_AR288_5 <- sub("\\.wav$", "_140.wav", AR_CD288)
OUT.PATH_AR288_5 <- file.path("New files", basename(FILE.OUT_AR288_5))
writeWave(CUT_AR288_5, extensible = FALSE, filename = OUT.PATH_AR288_5[i])
}, silent = TRUE)
if (inherits(ret, "try-error")) {
# do or log something
}
}
I have written the following script that fetches friends of Twitter users ("barackobama" in this example) in batches of 75,000 (5000 friends per API call x 15 API calls) every 15 minutes using rtweet. However, after the script is done running, I find that the friend ids repeat after a fixed interval. For instance, rows 1, 280001, and 560001 have the same ID. Rows 2, 280002, and 560002 have the same ID, and so on. I'm wondering if I'm understanding next_cursor in the API incorrectly.
u = "barackobama"
n_friends = lookup_users(u)$friends_count
curr_page = -1
fetched_friends = 0
i = 0
all_friends = NULL
while(fetched_friends < n_friends) {
if(rate_limit("get_friends")$remaining == 0) {
print(paste0("API limit reached. Reseting at ", rate_limit("get_friends")$reset_at))
Sys.sleep(as.numeric((rate_limit("get_friends")$reset + 0.1) * 60))
}
curr_friends = get_friends(u, n = 5000, retryonratelimit = TRUE, page = curr_page)
i = i + 1
all_friends = rbind(all_friends, curr_friends)
fetched_friends = nrow(all_friends)
print(paste0(i, ". ", fetched_friends, " out of ", n_friends, " fetched."))
curr_page = next_cursor(curr_friends)
}
Any help will be appreciated.
You are not doing anything wrong. From the documentation:
this ordering is subject to unannounced change and eventual consistency issues
For very large lists, the API simply won't return all the information you want.
I've wrote the following code:
require 'nn'
require 'cunn'
file = torch.DiskFile('train200.data', 'r')
size = file:readInt()
inputSize = file:readInt()
outputSize = file:readInt()
dataset = {}
function dataset:size() return size end;
for i=1,dataset:size() do
local input = torch.Tensor(inputSize)
for j=1,inputSize do
input[j] = file:readFloat()
end
local output = torch.Tensor(outputSize)
for j=1,outputSize do
output[j] = file:readFloat()
end
dataset[i] = {input:cuda(), output:cuda()}
end
net = nn.Sequential()
hiddenSize = inputSize * 2
net:add(nn.Linear(inputSize, hiddenSize))
net:add(nn.Tanh())
net:add(nn.Linear(hiddenSize, hiddenSize))
net:add(nn.Tanh())
net:add(nn.Linear(hiddenSize, outputSize))
criterion = nn.MSECriterion()
net = net:cuda()
criterion = criterion:cuda()
trainer = nn.StochasticGradient(net, criterion)
trainer.learningRate = 0.02
trainer.maxIteration = 100
trainer:train(dataset)
And it must works good (At least I think so), and it works correct when inputSize = 20. But when inputSize = 200 current error always is nan. At first I've thought that file reading part is incorrect. I've recheck it some times but it is working great. Also I found that sometimes too small or too big learning rate may affect on it. I've tried learning rate from 0.00001 up to 0.8, but still the same result. What I'm doing wrong?
Thanks,
Igor
This was an administrative issue. It was claimed that it was our fault, and our code was broken, when the issue was elsewhere in different code that can update what ours imports. So, this is a non-problem.
We are adding to values together that are in a dictionary and storing them in a variable to then be added to the NDB datastore.
Essentially, we are importing a csv and parsing the data. If the sum of two of the values is greater than 1,000.00, the float is changed to the thousands value.
Examples:
1263.13 would change to 1
51367.42 would change to 51
218.12 would be unchanged (218.12)
We think it is an issue with storing the data, but we are having trouble reproducing the the issue, and have verified that with a small amount of data, there will be "false positives". IE: there not being an issue.
Reading csv from blob:
blob_reader = blobstore.BlobReader(blob_info.key())
raw_blob = blob_reader.read()
result = [row for row in csv.DictReader(StringIO.StringIO(raw_blob), delimiter=',')]
# loop through all rows in the data/csv
for v, row in enumerate(result):
if set_data_type(row, v) == False:
return False
else:
length = len(result) + 1
# check values for type
if check_values(row, v, length) == False:
return False
else:
# calculate numerical data for storage
calculate_vars(row, v)
if not error_log:
# if no errors, store the row in the datastore
store_data(row, v)
# if there are no errors, add to transaction log.
if not error_log:
added_records = len(result)
old_new = "Upload Success - " + str(added_records) + " records added."
transactions(module, 'Datastore', data[0], 'CSV Bulk Upload', old_new, old_new)
Calculating values:
def calculate_vars(row, v):
# calcuate values for storage based on
# data passed from the csv
try:
float_sum = float(row['float1']) + float(row['float2'])
except Exception, e:
catch_error(["Value calculation error in row ", v+1, " Error: ", e])
return False
else:
# global var
calculated_val = float_sum
Storing Value:
def store_data(row, v):
try:
entry = Datastore(
float_sum = calculated_val
)
except Exception, e:
catch_error(["StoreDataError in row ", v, e])
return False
else:
# add entry to datastore.
entry.put()
The Datastore column 'float_sum' is an ndb.FloatProperty()
What we're trying to figure out is: what could cause the floats to be truncated to the thousands digits and nothing else when the number is larger than 1,000. Obviously, we are aware that adding two floats together will not cause this normally, which is why we think it could be an issue with the datastore itself.
An example of what I desire:
local X = {["Alpha"] = 5, ["Beta"] = this.Alpha+3}
print(X.Beta) --> error: [string "stdin"]:1: attempt to index global 'this' (a nil value)
is there a way to get this working, or a substitute I can use without too much code bloat(I want it to look presentable, so fenv hacks are out of the picture)
if anyone wants to take a crack at lua, repl.it is a good testing webpage for quick scripts
No there is no way to do this because the table does not yet exist and there is no notion of "self" in Lua (except via syntactic sugar for table methods). You have to do it in two steps:
local X = {["Alpha"] = 5}
X["Beta"] = X.Alpha+3
Note that you only need the square brackets if your key is not a string or if it is a string with characters other than any of [a-z][A-Z][0-9]_.
local X = {Alpha = 5}
X.Beta = X.Alpha+3
Update:
Based on what I saw on your pastebin, you probably should do this slightly differently:
local Alpha = 5
local X = {
Alpha = Alpha,
Beta = Alpha+3,
Gamma = someFunction(Alpha),
Eta = Alpha:method()
}
(obviously Alpha has no method because in the example it is a number but you get the idea, just wanted to show if Alpha were an object).