if I fetch a render of previous 1h (cache disabled), I get all the datapoints right up to current time. unaggregated here, showing 1 update per second. eg,
http://graphite/render?target=local.metric.path&from=-1h&format=json&cacheTimeout=0
... [-2960.12, 1505975193], [-2960.12, 1505975194], [-2960.12, 1505975195], [-2960.12, 1505975196], [-2960.12, 1505975197], [-2960.12, 1505975198], [-2960.12, 1505975199], [-2960.12, 1505975200], [-2960.12, 1505975201], [null, 1505975202]]}]
great. but if I now up the render to previous 2h, can now see it aggregating the data at 5s, and the last few points are all 'null'.
... [-2775.75, 1505975390], [-2667.612, 1505975395], [-2595.52, 1505975400], [-2595.52, 1505975405], [-2595.52, 1505975410], [-2595.52, 1505975415], [-2595.52, 1505975420], [-2595.52, 1505975425], [-2595.52, 1505975430], [-2595.52, 1505975435], [null, 1505975440], [null, 1505975445], [null, 1505975450], [null, 1505975455], [null, 1505975460], [null, 1505975465], [null, 1505975470], [null, 1505975475], [null, 1505975480], [null, 1505975485], [null, 1505975490], [null, 1505975495], [null, 1505975500], [null, 1505975505], [null, 1505975510], [null, 1505975515], [null, 1505975520], [null, 1505975525], [null, 1505975530], [null, 1505975535], [null, 1505975540], [null, 1505975545], [null, 1505975550], [null, 1505975555], [null, 1505975560], [null, 1505975565], [null, 1505975570], [null, 1505975575]]}]
further digging, the null points are all those after the last time the metric was written to whisper file on disk.
have tried looking at common causes..
metric updates are 1 per second, so as high as the max retention rate of the whisper files.
xFilesFactor is 0.5, but tried amending this to 0 with whisper-resize, and no change.
we're running graphite-web 0.9.15 from epel on CentOS 7.2
whisper info looks like..
[root#graphite]# whisper-info mymetric.wsp
maxRetention: 157680000
xFilesFactor: 0.0
aggregationMethod: average
fileSize: 1176592
Archive 0
retention: 3600
secondsPerPoint: 1
points: 3600
size: 43200
offset: 112
Archive 1
retention: 43200
secondsPerPoint: 5
points: 8640
size: 103680
offset: 43312
...
any ideas appreciated.
thanks,
Update: adding carbon.conf aggregate/cache settings..
[cache]
MAX_CACHE_SIZE = inf
CACHE_QUERY_INTERFACE = 0.0.0.0
CACHE_QUERY_PORT = 7002
LOG_CACHE_HITS = False
LOG_CACHE_QUEUE_SORTS = True
CACHE_WRITE_STRATEGY = sorted
[aggregator]
MAX_AGGREGATION_INTERVALS = 5
Queries that access more than one Whisper "retention archive" will/may return NULL values.
For example, let's assume the /opt/graphite/conf/storage-schemas.conf says:
[default]
pattern = .*
retentions = 60s:1d,10m:1y
and you get values when querying for the last 1 day. Querying for the last 2 days will/may result in NULL values for the last 1 day!
Also beware that when changing the retention configuration, existing archives should be resized accordingly:
(assuming the new configuration is 1m:400d 10m:3y, to reduce the chances of accidentally accessing multiple "retention archives")
cd /opt/graphite/storage/whisper
# see the old size of the data store
du -skh
find ./ -type f -name '*.wsp' -exec whisper-resize.py --nobackup {} 1m:400d 10m:3y \;
# see the new size of the data store (Whisper preallocates what space it needs)
du -skh
EDIT:
I haven't tested it, but whether the NULL problem will actually occur, probably has to do with how sparse the actual data points are, vs the xFilesFactor in storage-aggregation.conf (doc) changing which probably also requires running whisper-resize.py with --xFilesFactor=new_value
If the new_value is 0.0, accessing multiple "retention archives" from single query should work just fine.
xFilesFactor should be a floating point number between 0 and 1, and specifies what fraction of the previous retention level’s slots must have non-null values in order to aggregate to a non-null value. The default is 0.5.
Related
working on an economic optimization problem with pyomo, I would like to add a constraint to prevent the product of the commodity quantity and its price to go below zero (<0), avoiding a negative revenue. It appears that all the data are in a dataframe and I can't setup a constraint like:
def positive_revenue(model, t)
return model.P * model.C >=0
model.positive_rev = Constraint(model.T, rule=positive_revenue)
The system returns the error that the price is a scalar and it cannot process it. Indeed the price is set as such in the model:
model.T = Set(doc='quarter of year', initialize=df.quarter.tolist(), ordered=True)
model.P = Param(initialize=df.price.tolist(), doc='Price for each quarter')
##while the commodity is:
model.C = Var(model.T, domain=NonNegativeReals)
I just would like to apply that for each timestep (quarter of hour here) that:
price(t) * model.C(t) >=0
Can someone help me to spot the issue ? Thanks
Here are more information:
df dataframe:
df time_stamp price Status imbalance
quarter
0 2021-01-01 00:00:00 64.84 Final 16
1 2021-01-01 00:15:00 13.96 Final 38
2 2021-01-01 00:30:00 12.40 Final 46
index = quarter from 0 till 35049, so it is ok
Here is the df.info()
# Column Non-Null Count Dtype
--- ------ -------------- -----
0 time_stamp 35040 non-null datetime64[ns]
1 price 35040 non-null float64
2 Status 35040 non-null object
3 imbalance 35040 non-null int64
I modified the to_list() > to_dict() in model.T but still facing the same issue:
KeyError: "Cannot treat the scalar component 'P' as an indexed component" at the time model.T is defined in the model parameter, set and variables.
Here is the constraint where the system issues the error:
def revenue_positive(model,t):
for t in model.T:
return (model.C[t] * model.P[t]) >= 0
model.positive_revenue = Constraint(model.T,rule=revenue_positive)
Can't figure it out...any idea ?
UPDATE
Model works after dropping an unfortunate 'quarter' column somewhere...after I renamed the index as quarter.
It runs but i still get negative revenues, so the constraints seems not working at present, here is how it is written:
def revenue_positive(model,t):
for t in model.T:
return (model.C[t] * model.P[t]) >= 0
model.positive_revenue = Constraint(model.T,rule=revenue_positive)
What am I missing here ? Thanks for help, just beginning
Welcome to the site.
The problem you appear to be having is that you are not building your model parameter model.P as an indexed component. I believe you likely want it to be indexed by your set model.T.
When you make indexed params in pyomo you need to initialize it with some key:value pairing, like a python dictionary. You can make that from your data frame by re-indexing your data frame so that the quarter labels are the index values.
Caution: The construction you have for model.T and this assume there are no duplicates in the quarter names.
If you have duplicates (or get a warning) then you'll need to do something else. If the quarter labels are unique you can do this:
import pandas as pd
import pyomo.environ as pyo
df = pd.DataFrame({'qtr':['Q5', 'Q6', 'Q7'], 'price':[12.80, 11.50, 8.12]})
df.set_index('qtr', inplace=True)
print(df)
m = pyo.ConcreteModel()
m.T = pyo.Set(initialize=df.index.to_list())
m.price = pyo.Param(m.T, initialize=df['price'].to_dict())
m.pprint()
which should get you:
price
qtr
Q5 12.80
Q6 11.50
Q7 8.12
1 Set Declarations
T : Size=1, Index=None, Ordered=Insertion
Key : Dimen : Domain : Size : Members
None : 1 : Any : 3 : {'Q5', 'Q6', 'Q7'}
1 Param Declarations
price : Size=3, Index=T, Domain=Any, Default=None, Mutable=False
Key : Value
Q5 : 12.8
Q6 : 11.5
Q7 : 8.12
2 Declarations: T price
edit for clarity...
NOTE:
The first argument when you create a pyomo parameter is the indexing set. If this is not provided, pyomo assumes that it is a scalar. You are missing the set as shown in my example and highlighted with arrow here: :)
|
|
|
V
m.price = pyo.Param(m.T, initialize=df['price'].to_dict())
Also note, you will need to initialize model.P with a dictionary as I have in the example, not a list.
I have created a table with a collection. Inserted a record and took sstabledump of it and seeing there is range tombstone for it in the sstable. Does this tombstone ever get removed? Also when I run sstablemetadata on the only sstable, it shows "Estimated droppable tombstones" as 0.5", Similarly it shows one record with epoch time as insert time for - "Estimated tombstone drop times: 1548384720: 1". Does it mean that when I do sstablemetadata on a table having collections, the estimated droppable tombstone ratio and drop times values are not true and dependable values due to collection/list range tombstones?
CREATE TABLE ks.nmtest (
reservation_id text,
order_id text,
c1 int,
order_details map<text, text>,
PRIMARY KEY (reservation_id, order_id)
) WITH CLUSTERING ORDER BY (order_id ASC)
user#cqlsh:ks> insert into nmtest (reservation_id , order_id , c1, order_details ) values('3','3',3,{'key':'value'});
user#cqlsh:ks> select * from nmtest ;
reservation_id | order_id | c1 | order_details
----------------+----------+----+------------------
3 | 3 | 3 | {'key': 'value'}
(1 rows)
[root#localhost nmtest-e1302500201d11e983bb693c02c04c62]# sstabledump mc-5-big-Data.db
WARN 02:52:19,596 memtable_cleanup_threshold has been deprecated and should be removed from cassandra.yaml
[
{
"partition" : {
"key" : [ "3" ],
"position" : 0
},
"rows" : [
{
"type" : "row",
"position" : 41,
"clustering" : [ "3" ],
"liveness_info" : { "tstamp" : "2019-01-25T02:51:13.574409Z" },
"cells" : [
{ "name" : "c1", "value" : 3 },
{ "name" : "order_details", "deletion_info" : { "marked_deleted" : "2019-01-25T02:51:13.574408Z", "local_delete_time" : "2019-01-25T02:51:13Z" } },
{ "name" : "order_details", "path" : [ "key" ], "value" : "value" }
]
}
]
}
SSTable: /data/data/ks/nmtest-e1302500201d11e983bb693c02c04c62/mc-5-big
Partitioner: org.apache.cassandra.dht.Murmur3Partitioner
Bloom Filter FP chance: 0.010000
Minimum timestamp: 1548384673574408
Maximum timestamp: 1548384673574409
SSTable min local deletion time: 1548384673
SSTable max local deletion time: 2147483647
Compressor: org.apache.cassandra.io.compress.LZ4Compressor
Compression ratio: 1.0714285714285714
TTL min: 0
TTL max: 0
First token: -155496620801056360 (key=3)
Last token: -155496620801056360 (key=3)
minClustringValues: [3]
maxClustringValues: [3]
Estimated droppable tombstones: 0.5
SSTable Level: 0
Repaired at: 0
Replay positions covered: {CommitLogPosition(segmentId=1548382769966, position=6243201)=CommitLogPosition(segmentId=1548382769966, position=6433666)}
totalColumnsSet: 2
totalRows: 1
Estimated tombstone drop times:
1548384720: 1
Another quuestion was on the nodetool tablestats output - what does slice refer to in cassandra?
Average live cells per slice (last five minutes): 1.0
Maximum live cells per slice (last five minutes): 1
Average tombstones per slice (last five minutes): 1.0
Maximum tombstones per slice (last five minutes): 1
Dropped Mutations: 0
sstablemetadata does not have the information about your table that is not held within the sstable as it is not guaranteed to be run on system that has Cassandra running, and even if it was its very complex to be able to know how to pull the schema information from it.
Since the gc_grace_seconds is a table parameter and not in the metadata it defaults to assuming a 0 gc grace so the droppable times listed in that histogram will be more a histogram of the tombstone creation times by default. If you know your gc grace you can add it as a -g parameter to your sstablemetadata call. like:
sstablemetadata -g 864000 mc-5-big-Data.db
see http://cassandra.apache.org/doc/latest/tools/sstable/sstablemetadata.html for information on the tools output.
With collections it's just normal range tombstone with all that it entails. They are used to prevent the requirement of a read-before-write when overwriting the value of a multicell collection.
The Program aims at checking whether a student has taken a specified list of courses or not. But it displays false even if the student has taken that list of courses. What am I doing wrong?
Code:
%Structure Of facts
%student(ID,FirstName,LastName,Advisor,CreditsPassed,GPA,CourseTakenList).
%courses_taken(CourseId,CourseTitle, Credit,Grade, NoOfTimesRepeated).
%Fact
student(20135639,'Sara','Haider','Mr. Hussain Al-Arrayed',98,3.95,
[courses_taken('ITCE418', 'PRODUCTIVITY WITH IS TECHNOLOGY',3,'A', 0),
courses_taken('MATHS101', 'CALCULUS I', 3,'A', 0),
courses_taken('ACC112', 'FINANCIAL ACCOUNTING I', 3, 'A', 0),
courses_taken('ECON140', 'MICROECONOMICS', 3,'A', 0),
courses_taken('ENGL219', 'TECHNICAL REPORT WRITING', 3,'A', 0) ] ).
check_prereq([],Id):- !.
check_prereq([course(P)]|T,Id):-
student(Id,_,_,_,_,_, CoursesTakenList),
member( courses_taken(P,_,_,_,_), CoursesTakenList),
check_prereq(T,Id).
The following query should return true, since the student has taken those courses
?- check_prereq([course('ITCE418'), course('ACC112')],20135639).
false.
Am I doing something wrong?
You need to replace : [course(P)]|T with [course(P)|T]
check_prereq([],_):- !.
check_prereq([course(P)|T],Id):-
student(Id,_,_,_,_,_, CoursesTakenList),
member( courses_taken(P,_,_,_,_), CoursesTakenList),
check_prereq(T,Id).
Also in the first line replaced Id with _ (anonymous variable) in order to get rid of warnings(Singleton variable). Now it is working fine:
?- check_prereq([course('ITCE418'), course('ACC112')],20135639).
true ;
false.
Note that it returns true and asking for more solutions returns false but it succeeds as we expected.
so I wanted to start playing with it and test it, so i put this config:
storage-schemas.conf:
[short2]
pattern = ^short2\.
retentions = 10s:1m
storage-aggregation.conf
[sum]
pattern = \.count$
xFilesFactor = 0
aggregationMethod = sum
what I think that my config say:
get data every 10 seconds and save it to 1 minutes so total of 10 points will be saved
now if i go to
http://localhost/render/?target=short2.sum&format=json&from=-1h
I see many data with null values a lot more than 10,
ok so I give up on that, than I said let's try to feed it data once every 10 seconds, if i do
echo "short2.sum 22 `date +%s`" | nc -q0 127.0.0.1 2003
wait 11 seconds
echo "short2.sum 23 `date +%s`" | nc -q0 127.0.0.1 2003
now looking at the api I can see only the last point get registerd like:
[
null,
1464781920
],
[
null,
1464781980
],
[
null,
1464782040
],
[
null,
1464782100
],
[
23,
1464782160
],
now if I send it another point (a lot after 10 seconds)
echo "short2.sum 24 `date +%s`" | nc -q0 127.0.0.1 2003
this is what I get:
[
null,
1464781920
],
[
null,
1464781980
],
[
null,
1464782040
],
[
null,
1464782100
],
[
24,
1464782160
],
only once in a couple of tries I will see them count as new but they just overwriting each other instead of acting like new data
Actually:
[short2]
pattern = ^short2\.
retentions = 10s:1m
means: all metrics starts with short2. keep for 1 minute with 10 second resolution (each datapoint represents 10s). It also means if there are not defined other storage schemas for short2., it will have value only for 1 (last) minute.
http://graphite.readthedocs.io/en/latest/config-carbon.html#storage-schemas-conf
I can't find the correct mathematical formula to compute a SLA (availability) with Grafana:
I have graph to show the duration of downtime for each days:
From this, i would like to compute the SLA (eg: 99,5%).
On the graph for the selected period (Last 7 days) i can to have this data:
71258 is the sum of duration of downtime in second. I have this with summarize(1day, max, false)
I need to have the sum of duration of time for the selected period (here 7 days = 604800second). But how ?
If i have this last data, after i will do :
(100% * 604800) / 71258 = X %
100% - X % = My SLA!!
My question is: Which formula use to have the duration for a selected period in Grafana ?
One of the database you can run behind Grafana, is Axibase Time Series Database (ATSD). It provides built-in aggregation functions that can perform SLA-type calculations, for example, to compute % of the period when the value exceeded the threshold.
THRESHOLD_COUNT - number of violations in the period
THRESHOLD_DURATION - cumulative duration of the violations
THRESHOLD_PERCENT - duration divided by period
In your example, that would be THRESHOLD_PERCENT.
Here's a sample SLA report for Amazon Web Services instance: https://apps.axibase.com/chartlab/0aa34311/6/. THRESHOLD_PERCENT is visualized on the top chart.
The API request looks as follows:
{
"queries": [{
"startDate": "2016-02-22T00:00:00Z",
"endDate": "2016-02-23T00:00:00Z",
"timeFormat": "iso",
"entity": "nurswgvml007",
"metric": "app.response_time",
"aggregate": {
"types": [
"THRESHOLD_COUNT",
"THRESHOLD_DURATION",
"THRESHOLD_PERCENT"
],
"period": {
"count": 1,
"unit": "HOUR"
},
"threshold": {
"max": 200
}
}
}]
}
ATSD driver: https://github.com/grafana/grafana-plugins/tree/master/datasources/atsd
Disclosure: I work for Axibase.