Grafana: only 1 minute of data points visible for last 5 minutes - graphite

I cannot seem to determine why Grafana is only displaying data points for the past minute even though we have been collecting data for longer than that (couple days).
That is, switching to "5 minutes ago" only displays the last minute of data and seems to cut off with each refresh. However, selecting a specific time range displays all data points correctly.
For this data path, storage-schemas.conf is setup like so:
retentions = 1s:24h,15s:7d,1m:30d
From what I gather, that means that I should have 1-second precision data for the past 24 hours, 15-second data for the past 7 days, and 1-minute average for 30 days. Is this also a correct assumption and if not, could it be related to the grafana problem?

Related

Assigning task to resource with increased working hours and reduced availability

I am facing a problem when I am assigning a one day task to a resource who works for 9hrs/day but with 25% availability. Following are the steps
Created a calendar with 9hrs as working hours (for all days) as shown below
Assigned availability to "Dummy" Resource as shown below
Created a 1 day task "Dummy" as shown below
Problem is
Work Column must be 2.25 hrs
Calculation: Dummy resource for 9hrs/day at 25% availability = (25/100)*9 hrs = 2.25hrs
but it is showing is 2.13 hrs
How to assign a long duration task (say 1 year) to above resource with different availability (as shown in image above). [Note: There are other resource who work for 8hrs/day with different availibility hence I cannot change MS Project "OPTIONS"]
I have created 3 calendars - CAL_1, CAL_2 and CAL_3 for 3 different locations. Each location has its own non-working days in each quarter. For example 15 weekdays are not working in Mar, Jun, Sept, Dec in CAL_1 every year. 10 weekdays in same manner for CAL_2 every year and so on. Is there any VBA script by which I can automate such work for every year (note: weekends are not included).

How can I fetch a datapoint at certain time every day in Graphite?

I have a series of data that increases by time and resets to zero at 18:00 every day. How can I make a Graphite plot that only contains datapoints at 17:59 in the last 30 days?
I have tried summarize(1d, max, false), but it by default bins data into buckets that are calculated by rounding to the nearest interval to current time. So I cannot specify the beginning time of each bucket to be 18:00.
I couldn't find anything that exactly matches what you want. There are functions like timeSlice and timeStack but they do not really fit.
An alternative is to use the graphite function nonNegativeDerivative. It ignores when counters are reset to zero and only shows counter increments.

Sending data with static time to Graphite

I am currently sending logs to Graphite that are updated with timestamps and a value every 30 minutes, however I am also keeping logs that calculate trends in this data for every 30 minute period over each day in the course of a week.
For example, in JSON:
{
"stat_id": "1-0-32",
"_t": "16:00",
"value": 26.0,
"value_count": 4.0,
"day_type": "Weekday",
"day": "Monday"
}
would be the average value for 16:00 on a Monday for a specific metric. I would like to be able to graph this trend data on top of the raw data that is already being graphed. I've already used Graphite functions like averageSeries(timeStack(metric, "7d", x, y)) to try to recreate the trend data but am not confident it represents the trends as accurately and completely.
So I guess the question is can I do this by sending data to Graphite that only changes in value and where the timestamps are static and somehow represents hours in the week?
What I ended up doing is adding the trend data as if it were real-time data by applying the time stamps to the current day, making sure not to add future metrics.

Duration and work concerning tasks in MS Project

i have a problem in MS Project 2013:
i´d like to set up a work package that has a certain duration but in this duration there is no work like 8 hours x (duration in days) required but e.g. 5 days of work in a timeframe of 30 days.
this is annoying to set up but it works. now ms project does not draw the respective bar for 30 days of duration but only for 5 days (the work). at the end of the bar a little arrows goes on for another 25 days (resulting in 5+25=30days) and then turns down to the next work package. i´d rather like to see a bar for the full 30 days and then for the arrow to go down.
is there a way to do this?
it seems wrong that the bar depends on the work rather than the duration because it is not likely that the work gets done in the first 5 days and then you just wait for 25 days.
Create a new task and set the Duration to 30d and make sure the type is "Fixed Duration" and set the task's Work to 40h.

How to download intraday stock market data with R

All,
I'm looking to download stock data either from Yahoo or Google on 15 - 60 minute intervals for as much history as I can get. I've come up with a crude solution as follows:
library(RCurl)
tmp <- getURL('https://www.google.com/finance/getprices?i=900&p=1000d&f=d,o,h,l,c,v&df=cpct&q=AAPL')
tmp <- strsplit(tmp,'\n')
tmp <- tmp[[1]]
tmp <- tmp[-c(1:8)]
tmp <- strsplit(tmp,',')
tmp <- do.call('rbind',tmp)
tmp <- apply(tmp,2,as.numeric)
tmp <- tmp[-apply(tmp,1,function(x) any(is.na(x))),]
Given the amount of data I'm looking to import, I worry that this could be computationally expensive. I also don't for the life of me, understand how the time stamps are coded in Yahoo and Google.
So my question is twofold--what's a simple, elegant way to quickly ingest data for a series of stocks into R, and how do I interpret the time stamping on the Google/Yahoo files that I would be using?
I will try to answer timestamp question first. Please note this is my interpretation and I could be wrong.
Using the link in your example https://www.google.com/finance/getprices?i=900&p=1000d&f=d,o,h,l,c,v&df=cpct&q=AAPL I get following data :
EXCHANGE%3DNASDAQ
MARKET_OPEN_MINUTE=570
MARKET_CLOSE_MINUTE=960
INTERVAL=900
COLUMNS=DATE,CLOSE,HIGH,LOW,OPEN,VOLUME
DATA=
TIMEZONE_OFFSET=-300
a1357828200,528.5999,528.62,528.14,528.55,129259
1,522.63,528.72,522,528.6499,2054578
2,523.11,523.69,520.75,522.77,1422586
3,520.48,523.11,519.6501,523.09,1130409
4,518.28,520.579,517.86,520.34,1215466
5,518.8501,519.48,517.33,517.94,832100
6,518.685,520.22,518.63,518.85,565411
7,516.55,519.2,516.55,518.64,617281
...
...
Note the first value of first column a1357828200, my intuition was that this has something to do with POSIXct. Hence a quick check :
> as.POSIXct(1357828200, origin = '1970-01-01', tz='EST')
[1] "2013-01-10 14:30:00 EST"
So my intuition seems to be correct. But the time seems to be off. Now we have one more info in the data. TIMEZONE_OFFSET=-300. So if we offset our timestamps by this amount we should get :
as.POSIXct(1357828200-300*60, origin = '1970-01-01', tz='EST')
[1] "2013-01-10 09:30:00 EST"
Note that I didn't know which day data you had requested. But quick check on google finance reveals, those were indeed price levels on 10th Jan 2013.
Remaining values from first column seem to be some sort of offset from first row value.
So downloading and standardizing the data ended up being more much of a bear than I figured it would--about 150 lines of code. The problem is that while Google provides the past 50 training days of data for all exchange-traded stocks, the time stamps within the days are not standardized: an index of '1,' for example could either refer to the first of second time increment on the first trading day in the data set. Even worse, stocks that only trade at low volumes only have entries where a transaction is recorded. For a high-volume stock like APPL that's no problem, but for low-volume small caps it means that your series will be missing much if not the majority of the data. This was problematic because I need all the stock series to lie neatly on to of each other for the analysis I'm doing.
Fortunately, there is still a general structure to the data. Using this link:
https://www.google.com/finance/getprices?i=1800&p=1000d&f=d,o,h,l,c,v&df=cpct&q=AAPL
and changing the stock ticker at the end will give you the past 50 days of trading days on 1/2-hourly increment. POSIX time stamps, very helpfully decoded by #geektrader, appear in the timestamp column at 3-week intervals. Though the timestamp indexes don't invariably correspond in a convenient 1:1 manner (I almost suspect this was intentional on Google's part) there is a pattern. For example, for the half-hourly series that I looked at the first trading day of ever three-week increment uniformly has timestamp indexes running in the 1:15 neighborhood. This could be 1:13, 1:14, 2:15--it all depends on the stock. I'm not sure what the 14th and 15th entries are: I suspect they are either daily summaries or after-hours trading info. The point is that there's no consistent pattern you can bank on.The first stamp in a training day, sadly, does not always contain the opening data. Same thing for the last entry and the closing data. I found that the only way to know what actually represents the trading data is to compare the numbers to the series on Google maps. After days of futiley trying to figure out how to pry a 1:1 mapping patter from the data, I settled on a "ballpark" strategy. I scraped APPL's data (a very high-volume traded stock) and set its timestamp indexes within each trading day as the reference values for the entire market. All days had a minimum of 13 increments, corresponding to the 6.5 hour trading day, but some had 14 or 15. Where this was the case I just truncated by taking the first 13 indexes. From there I used a while loop to essentially progress through the downloaded data of each stock ticker and compare its time stamp indexes within a given training day to the APPL timestamps. I kept the overlap, gap-filled the missing data, and cut out the non-overlapping portions.
Sounds like a simple fix, but for low-volume stocks with sparse transaction data there were literally dozens of special cases that I had to bake in and lots of data to interpolate. I got some pretty bizarre results for some of these that I know are incorrect. For high-volume, mid- and large-cap stocks, however, the solution worked brilliantly: for the most part the series either synced up very neatly with the APPL data and matched their Google Finance profiles perfectly.
There's no way around the fact that this method introduces some error, and I still need to fine-tune the method for spare small-caps. That said, shifting a series by a half hour or gap-filling a single time increment introduces a very minor amount of error relative to the overall movement of the market and the stock. I am confident that this data set I have is "good enough" to allow me to get relevant answers to some questions that I have. Getting this stuff commercially costs literally thousands of dollars.
Thoughts or suggestions?
Why not loading the data from Quandl? E.g.
library(Quandl)
Quandl('YAHOO/AAPL')
Update: sorry, I have just realized that only daily data is fetched with Quandl - but I leave my answer here as Quandl is really easy to query in similar cases
For the timezone offset, try:
as.POSIXct(1357828200, origin = '1970-01-01', tz=Sys.timezone(location = TRUE))
(The tz will automatically adjust according to your location)

Resources