Extend dates in frequency table - r

I have different data sets that look like the following:
structure(c(1414406460, 1414635660, 1414636260, 1414636920, 1414637040,
1414639020, 1414711140, 1414714500, 1414718760, 1414718820, 1414727520,
1414727640, 1414727760, 1414898220, 1414898880, 1414899000, 1414899120,
1414899240, 1414899300, 1414899480, 1414899600, 1414900140, 1414900260,
1414900560, 1414900620, 1414900860, 1414901040, 1414919940, 1414920420,
1414951320, 1414971720, 1414977600, 1414977660, 1414978620, 1414984980,
1414988100, 1414988160, 1414989000, 1414989240, 1414989900, 1414990020,
1414990080, 1414990320, 1414990500, 1414990920, 1414991100, 1414991220,
1414991340, 1414991940, 1414992240, 1414992780, 1414992900, 1414993020,
1414993140, 1415001600, 1415001660, 1415001720, 1415001840, 1415001960,
1415002140, 1415002260, 1415003280, 1415003340, 1415018040, 1415040780,
1415040900, 1415040960, 1415041080, 1415041320, 1415041440, 1415041560,
1415041980, 1415042220, 1415042280, 1415042400, 1415042820, 1415043240,
1415043360, 1415043420, 1415043540, 1415043600, 1415043660, 1415043720,
1415043840, 1415043900, 1415044020, 1415044260, 1415044320, 1415044440,
1415044560, 1415044620, 1415044680, 1415044800, 1415044860, 1415044920,
1415044980, 1415045100, 1415045160, 1415045280, 1415045340, 1415045400,
1415045460, 1415045520, 1415045700, 1415045760, 1415045820, 1415045940,
1415046000, 1415046120, 1415046240, 1415046360, 1415046420, 1415046540,
1415046600, 1415046720, 1415046780, 1415046900, 1415047020, 1415047080,
1415047140, 1415047260, 1415047320, 1415047380, 1415047440, 1415047560,
1415047620, 1415047680, 1415047740, 1415047800, 1415047920, 1415048040,
1415048160, 1415048280, 1415048340, 1415048460, 1415048520, 1415048580,
1415048640, 1415048760, 1415048820, 1415048940, 1415049000, 1415049060,
1415049120, 1415049180, 1415049420, 1415049600, 1415049720, 1415049780,
1415049900, 1415049960, 1415050020, 1415050140, 1415050320, 1415050500,
1415050620, 1415050860, 1415050980, 1415051220, 1415051520, 1415051580,
1415051700, 1415051760, 1415051880, 1415051940, 1415052000, 1415052120,
1415052180, 1415052240, 1415052360, 1415052540, 1415052660, 1415052720,
1415052780, 1415052900, 1415053020, 1415053080, 1415053140, 1415053260,
1415053380, 1415053440, 1415053500, 1415053560, 1415053620, 1415053680,
1415053800, 1415053860, 1415053980, 1415054100, 1415054160, 1415054760,
1415054820, 1415055120, 1415055180, 1415055480, 1415056920, 1415057160,
1415057760, 1415057760, 1415058720, 1415067660, 1415067780, 1415067900,
1415068020, 1415068080, 1415068140, 1415068980, 1415069100, 1415069400,
1415069520, 1415069580, 1415070000, 1415070060, 1415075760, 1415076720,
1415076840, 1415077200, 1415077500, 1415077620, 1415082600, 1415082660,
1415083140, 1415083260, 1415083380, 1415083440, 1415083500, 1415083800,
1415084100, 1415136600, 1415141340, 1415142780, 1415212920, 1415304360,
1415319300, 1415319900, 1415320320, 1415329140, 1415337660, 1415338680,
1415338980, 1415339040, 1415339160, 1415348640, 1415348820, 1415348940,
1415349000, 1415349060, 1415349120, 1415349180, 1415371200, 1415371320,
1415371500, 1415371800, 1415371920, 1415372040, 1415372100, 1415372160,
1415372220, 1415372280, 1415372340, 1415384520, 1415384580, 1415384640,
1415391720, 1415391840, 1415396160, 1415396580, 1415396760, 1415396940,
1415397780, 1415398080, 1415398380, 1415398380, 1415413620, 1415413680,
1415413740, 1415413800, 1415413860, 1415414040, 1415414400, 1415421900,
1415461920, 1415472240, 1415479740, 1415482920, 1415483280, 1415508780,
1415509020, 1415509140, 1415509620, 1415509800, 1415513760, 1415513880,
1415513940, 1415521980, 1415522040, 1415522100, 1415542260, 1415546520,
1415566740, 1415566800, 1415568300, 1415568360, 1415581380, 1415581560,
1415581620, 1415595000, 1415595240, 1415595480, 1415595540, 1415595720,
1415596560, 1415597100, 1415598360, 1415598540, 1415598960, 1415609160,
1415639700, 1415639760, 1415639880, 1415656320, 1415664360, 1415664480,
1415667960, 1415668500, 1415668740, 1415671620, 1415686740, 1415733660,
1415744280, 1415753340, 1415833500, 1415833620, 1415863260, 1415920320,
1415927760, 1415929020, 1415929260, 1415929320, 1415929500, 1415929920,
1415938560, 1415938920, 1415939400, 1416018540, 1416018600, 1416090120,
1416090300, 1416090360, 1416090480, 1416099900, 1416188640, 1416188700,
1416189120, 1416189240, 1416635940, 1416636000, 1416638400, 1416638520,
1416639180, 1416702720, 1416811860, 1416811980, 1416812040, 1416875940,
1416876000, 1416876060, 1416977640, 1416978360, 1417047780, 1417047840,
1417066620, 1417066680, 1417219320, 1417221240, 1417221300, 1417221540,
1417221600, 1417222740, 1417226580, 1417226640, 1417226700, 1417240560,
1417300800, 1417301400, 1417307280, 1417307400, 1417314780, 1417314900,
1417484160, 1417489200, 1418166300, 1418166540, 1418280240, 1418342280,
1418342340, 1418703240, 1418703240, 1420096320, 1420761360, 1420761720,
1420761840, 1420762020, 1421724780, 1422230580, 1422238380, 1422238500,
1422238560, 1422238620, 1422506820, 1423182120, 1423273080, 1423273200,
1423273260, 1423355820, 1424655360, 1424657520, 1424661840, 1424661900,
1424671260, 1424832840, 1424839980, 1424840340, 1424841900, 1424842020,
1424842800, 1424843040, 1425436080, 1425436200, 1425436320, 1425438960,
1425439020, 1425959880, 1426120200, 1426996140, 1426996380, 1427074800,
1427078100, 1427334360, 1427334420, 1427334660, 1427587800, 1427676780,
1427676900, 1427676960, 1429924860, 1429925220, 1429925520, 1442449440,
1443917580, 1444026600, 1444026780, 1444085880, 1444091040, 1444113300,
1444122900, 1444432680, 1444462920, 1444462980, 1444463040, 1445056380,
1445057700, 1445142420, 1445219160, 1445224500), class = c("POSIXct",
"POSIXt"), tzone = "America/Anguilla")
Now I want to count the frequency of each date/time cobination with timesteps of 30 minutes. I do that with the following code:
timestep = "30 min"
det_interval = data.frame(table(cut(dateseq, breaks = timestep)))
But my study period starts earlier and is exactly one year. So how can I extend the frequency table so that it starts at "2014-10-21 00:00:00" and stops at "2015-10-21 23:59:59"?
I tried to make a separate time sequence, but then I have to merge it with the following DF:
timestep = "30 min"
start = "2014-10-21 00:00"
end = "2015-10-21 23:59"
timeseq = seq(from = as.POSIXct(start), to = as.POSIXct(end), by = timestep)


I think this should do it. You can pass a sequence of breaks to the cut(), instead of the step (which starts at the min of data and goes to the max of the vector).
timestep<-seq.POSIXt(strptime("2014-10-21 00:00:00","%Y-%m-%d %H:%M:%S"), strptime("2015-10-21 23:59:59","%Y-%m-%d %H:%M:%S"), timestep)
det_interval = data.frame(table(cut(dateseq, breaks = timestep, right=T, include.lowest=T)))
head(det_interval)
tail(det_interval)

Related

Determine start date within time series

I hope you're doing well! I have a theoretical time-series analysis problem I hope that you can help me sort out.
To start, you'll find a reproducible example of my dataset below. Date is in a daily timescale. Q25 is 25th or lower quartile of my non-transformed data, Q75 is the 75th or upper quartile of my non-transformed data, fit is local weighted fit of the median, firstder is the first derivative of fit, and secondder is the second derivative of fit.
Plotting out fit produces two oscillations and then a steady increase in the data. Plotting the quartiles around that produces a large spread of data that narrows towards the increase in fit data. The first derivative shows the rate of change of the fit and this is where my issue comes in. I'm not sure where the increase in fit data starts based on the first derivative data. Logically, I know the signal-to-noise start date has to occur after March 7 (based on the quartiles), and before March 20 (before the steady increase in fit data). And this is also represented in the first derivative for about the same interval where the negative-to-positive inflection point changes on March 5th, becomes positive on March 16th, and then produces a stationary time series.
All that being said, should my exact start date be the change in the inflection point from the first derivative, or be the first positive value on March 16th?
I appreciate your time in this problem and any thoughts you may have!
data<-structure(list(Date = structure(c(1485950474, 1486036874, 1486123274,
1486209674, 1486296074, 1486382474, 1486468874, 1486555274, 1486641674,
1486728074, 1486814474, 1486900874, 1486987274, 1487073674, 1487160074,
1487246474, 1487332874, 1487419274, 1487505674, 1487592074, 1487678474,
1487764874, 1487851274, 1487937674, 1488024074, 1488110474, 1488196874,
1488283274, 1488369674, 1488456074, 1488542474, 1488628874, 1488715274,
1488801674, 1488888074, 1488974474, 1489060874, 1489147274, 1489233674,
1489320074, 1489406474, 1489492874, 1489579274, 1489665674, 1489752074,
1489838474, 1489924874, 1490011274, 1490097674, 1490184074, 1490270474,
1490356874, 1490443274, 1490529674, 1490616074, 1490702474, 1490788874,
1490875274, 1490961674, 1491048074, 1491134474, 1491220874, 1491307274,
1491393674, 1491480074, 1491566474, 1491652874, 1491739274, 1491825674,
1491912074, 1491998474, 1492084874, 1492171274, 1492257674, 1492344074,
1492430474, 1492516874, 1492603274, 1492689674, 1492776074, 1492862474,
1492948874, 1493035274, 1493121674, 1493208074, 1493294474, 1493380874,
1493467274, 1493553674, 1493640074, 1493726474, 1493812874, 1493899274,
1493985674, 1494072074, 1494158474, 1494244874, 1494331274, 1494417674,
1494504074, 1494590474, 1494676874, 1494763274, 1494849674, 1494936074,
1495022474, 1495108874, 1495195274, 1495281674, 1495368074), tzone = "UTC", class = c("POSIXct",
"POSIXt")), Q25 = c(-1.61495132528742, -3.86616056128065, -3.92140420424278,
-4.8011229557052, -8.64427034627082, -3.11323607034871, -4.3673083843457,
-1.45023104534208, 0.395769745934938, -1.49394189431791, -3.54063822876105,
-4.36090193633662, -0.966958995958447, -2.43233048854294, -0.181367797683111,
0.826258942687981, 3.36833418895383, -6.8991417494414, -1.15773470862185,
-1.75360705873163, 1.83790453304777, 2.11575746130393, -3.82025172988123,
0.679651741170909, -4.64628184041103, -6.91923314565111, 0.550274303541761,
0.104011128328036, -0.895257855280075, -0.801630235696042, 2.27958927430356,
2.98003963398985, 3.41649824319921, 1.56559818977215, -2.20923132476973,
0.552658760232765, 0.15158829140461, -4.75454688546242, -0.595460561248954,
-2.53729443345183, -0.826010503400985, -5.20578683534568, -2.78364193219594,
-3.62503323095109, 3.37820215582788, -2.53645164034493, -1.76051141957494,
-1.0256290530567, 1.94178279643985, 0.261239031590387, 0.00321585342072063,
2.87814873140354, -2.26732156613212, 2.65097224867168, -4.16746046231376,
1.64816233695592, 3.50505415841016, 2.83685877611882, 1.66353660199615,
2.27900517713667, 5.47721995923733, -5.31044894311933, 7.30753839733595,
5.50143585044911, -1.25129055380416, -2.41051058119916, 3.69266303212359,
2.28752278841533, -0.275687673398348, 5.74597173218469, 6.5773422259343,
3.72096844335478, 2.05388534852328, 5.41063696868948, 0.526467452167141,
1.60445671702256, -1.80394989627014, 1.56432488418924, 5.95370989889123,
7.94953250403525, 4.09121878799004, 2.11516919787794, -2.12808005361608,
6.77215849921842, 9.53718510298556, 4.16562173164636, 10.4573226478082,
7.703077796612, 7.55811710979136, 4.47194951592662, 10.2104312432178,
11.3454383477984, 0.997649090931488, 4.84898050707927, 10.8819209584302,
8.06296236341084, 11.3317616787558, 7.51878628894305, 7.87729934765305,
11.9108509727303, 6.77401202490232, 5.36297357453455, 10.6362047038983,
8.68979831512869, 4.0465996534104, 11.9579904470733, 9.41141176380086,
10.5754750604254, 12.6944336852953, 7.61563466861022), Q75 = c(5.93775779359077,
26.4536084846094, 7.92690107568623, 16.195405687679, 3.47567054091916,
34.9690262666155, 15.5126126583077, 24.4425589002446, 29.7425859431597,
23.1420118192775, 26.827758017105, 18.6306368759596, 19.759179203689,
10.0667740183259, 30.9080218485755, 10.0628623899296, 21.1120424008512,
12.1232187464341, 14.9571040303508, 11.4927011052638, 16.1617172813173,
19.0606972964125, 8.39991659547325, 9.5080530252195, 10.2717546026802,
12.018391863395, 27.2666992661895, 12.5172584337237, 19.9658806224003,
6.90019918091751, 18.4119063276997, 23.2991253786256, 27.95161418973,
16.9477966472485, 26.3880458021082, 19.2178725103802, 5.58699033890406,
9.82525729279156, 6.22139350667344, 5.6625294221828, 8.18283315939774,
4.78856479855966, 4.91215612536983, 5.35278870440784, 15.7471499356884,
7.95473965312171, 7.58463611165082, 6.03119890210746, 9.88624343762245,
6.66377352843609, 6.92675024060609, 7.20403099201013, 6.96877369392089,
17.7034248870798, 6.22890341708267, 6.1624397247754, 23.3856864094132,
7.13518162203812, 6.96344109315883, 7.69414570220079, 18.0859103957135,
7.52300478408242, 10.1635801549871, 10.021556657451, 8.51746254314866,
7.83000625461296, 15.4938419153615, 8.6844260972191, 8.07596479745038,
13.1423674521087, 8.04161364299224, 10.7442773622841, 8.58410892324644,
9.08436532340561, 8.84748510783176, 9.27529549461203, 9.01978932806698,
9.99776533859531, 9.61123990151036, 11.2228855544025, 10.3285714984086,
10.7107229417799, 11.452541129334, 11.9951421202043, 11.3568792509498,
11.139621487692, 12.957244784325, 13.1010906952192, 11.8445972599726,
12.8124554609003, 12.1817389611984, 13.4529860098547, 13.1808997426024,
12.568956945967, 13.9405958892683, 14.4445923505263, 14.5816203429081,
12.798362023978, 13.7926596005317, 14.3284196983115, 14.3967490595795,
14.3699332949429, 13.8061418130819, 15.4045229902535, 15.328632395916,
15.5928587109464, 15.5111381098579, 15.7167488979248, 16.4121827249844,
16.7700564366026), fit = c(1.3157822724014, 1.44491806546299,
1.67963756121542, 1.96834398237369, 2.32222986513481, 2.73223146146706,
3.16143742264514, 3.74278329406317, 4.4673163398484, 5.08529278937518,
5.58735598987316, 6.01592790788482, 6.19893270175371, 6.0219082198616,
5.64253432163072, 5.29694818196536, 4.89670493804841, 4.35145910275626,
3.89449691453349, 3.48150649031492, 3.06858491643756, 2.88963188544926,
3.13399806321574, 3.62311989322663, 4.03902573446563, 4.40598627768245,
4.84291047423098, 5.1737840740012, 5.3972440468493, 5.5747020603732,
5.62430591107552, 5.42843052467024, 5.07513358262307, 4.79108701506415,
4.59907825712695, 4.39731440509327, 4.22559688081583, 4.10100609028878,
4.00444369172723, 3.92144298531529, 3.82259220819525, 3.72499526558926,
3.68395895980124, 3.69588308031619, 3.73924432798967, 3.84246487218137,
4.07884774763199, 4.41108295888359, 4.70167312999791, 4.95537881350854,
5.2206483181831, 5.42551590243433, 5.52148736399275, 5.55736071284688,
5.60710852579646, 5.65757759073701, 5.68911425674423, 5.76594044238814,
5.93786454015275, 6.15175825295678, 6.31743846502224, 6.40077523837882,
6.45704948591979, 6.53019436816257, 6.59356685208809, 6.63353784524384,
6.71356141899707, 6.88849022040772, 7.11437487009308, 7.30646639975639,
7.43724432723552, 7.55279324817994, 7.67877181101032, 7.76924002146674,
7.83161170884946, 7.97157625691941, 8.25223488219952, 8.60947602940562,
8.95816992458796, 9.34076728750423, 9.77554331222275, 10.1411049362597,
10.3842988541376, 10.5696053585185, 10.7520817841281, 10.9357595672387,
11.0970528791622, 11.2495931571849, 11.3764752236255, 11.4864715266717,
11.6317299424136, 11.8381584436134, 12.0667779318613, 12.2724056764894,
12.462010561811, 12.6517333832877, 12.8101492769744, 12.9055352762602,
12.9678598772259, 13.0582354099638, 13.1489397497677, 13.2204738414797,
13.346284619515, 13.6054940294766, 13.9436193637562, 14.2337005769519,
14.5449448398809, 14.8895799498019, 15.0551768009747, 15.0689572800127
), firstder = c(0.0542499277820437, 0.193160412687084, 0.264645386746196,
0.318230646770668, 0.390583391620104, 0.410606699200811, 0.484714112557398,
0.683182658658343, 0.699350916534123, 0.546311900646561, 0.476582322984034,
0.33921923563074, 0.000346679118119919, -0.32275830659655, -0.377372654859586,
-0.342379980870621, -0.492111485610006, -0.524917784293232, -0.414059192641829,
-0.430018688265099, -0.343482693656914, 0.0295127267198723, 0.42189373253822,
0.482044173095213, 0.364522990904745, 0.40991488301477, 0.40715895907959,
0.264020778627613, 0.200548459021332, 0.136695124879259, -0.0667758528503706,
-0.308783766357995, -0.344835056787729, -0.22338628389576, -0.19056674389956,
-0.195775242472453, -0.146360055189657, -0.107867992742261, -0.0856184200473131,
-0.0883963049921002, -0.106496806989568, -0.0747428483921662,
-0.0103234284849929, 0.028493059030597, 0.0620691939868203, 0.163240621281308,
0.304123951137378, 0.325609827601989, 0.261609166722046, 0.261432729205552,
0.249586474110962, 0.150199026157553, 0.0521536950613295, 0.0370628072573624,
0.0565243651980056, 0.0371337817771211, 0.0409727028064402, 0.124422569131023,
0.207609809433488, 0.20232516927351, 0.121600832063498, 0.058433044321534,
0.0638003776220697, 0.0745713396178918, 0.0471802520722933, 0.0467708263829785,
0.126045851395065, 0.213953247074989, 0.220308792495525, 0.158550331399022,
0.11422743390592, 0.123806714974779, 0.114997378074604, 0.0651990840907102,
0.0828996021118185, 0.210617558388392, 0.336478451788591, 0.356675237198802,
0.354610868118913, 0.419333862640583, 0.419974858146042, 0.301270480481834,
0.201419853206041, 0.17882844049566, 0.186628379656891, 0.172934534594114,
0.156583940148236, 0.142289490196014, 0.11234075824169, 0.119081439314575,
0.177295034391252, 0.226764155293772, 0.22057696671022, 0.193643700730051,
0.190744241252391, 0.181381161962744, 0.127949080877661, 0.0681406193708671,
0.0729227267768433, 0.0983314755975622, 0.0766175682172481, 0.0819140886989596,
0.188151474480757, 0.320764600927798, 0.32011829707578, 0.283266397091015,
0.351814702578002, 0.276441515194414, 0.0724489974588587, -0.0273030060468944
), secondder = c(0.172623240328004, 0.105197729482076, 0.0377722186361492,
0.0693983014127931, 0.0753071882860794, -0.0352605731246656,
0.18347539983784, 0.213461692364051, -0.181125176612492, -0.124952855162631,
-0.0145063001624228, -0.260219874544165, -0.417525238481075,
-0.228684732948264, 0.119456036422192, -0.0494706884442619, -0.249992321034509,
0.184379723668058, 0.037337459634748, -0.0692564508812885, 0.242328440097658,
0.503662400655915, 0.281099610980781, -0.160798729866795, -0.0742436345141417,
0.165027418734192, -0.170539266604553, -0.1157370942994, -0.0112075449131641,
-0.116499123370982, -0.290442832088276, -0.193572994926973, 0.121470414067507,
0.12142713171643, -0.055788051724031, 0.0453710545782453, 0.0534593199873461,
0.0235248049074466, 0.0209743404824492, -0.0265301103720232,
-0.00967089362291196, 0.0731788108177152, 0.0556600289966314,
0.0219729460345484, 0.0451793238778984, 0.157163530711077, 0.124603129001063,
-0.081631376071841, -0.0463699456880455, 0.0460170706550578,
-0.0697095808442372, -0.129065315062581, -0.0670253471298663,
0.0368435715219322, 0.0020795443593542, -0.0408607112011232,
0.0485385532597613, 0.118361179389404, 0.0480133012155273, -0.058582581535485,
-0.102866092884539, -0.0234694825993884, 0.0342041492004599,
-0.0126622252088158, -0.0421199498823812, 0.0413010985037516,
0.117248951520421, 0.0585658398394289, -0.045854748998357, -0.0776621731946507,
-0.0109836217915529, 0.0301421839292724, -0.0477608577296227,
-0.0518357302381656, 0.0872367662803821, 0.168199146272765, 0.0835226405276321,
-0.0431290697072093, 0.039000331547431, 0.0904456574959092, -0.0891636664849909,
-0.148245088843424, -0.0514561657081618, 0.00627334028739845,
0.00932653803506511, -0.036714228160621, 0.00401303926886598,
-0.0326019391733094, -0.0272955247353401, 0.0407768868811118,
0.0756503032722406, 0.0232879385327998, -0.0356623156999039,
-0.0182042162604343, 0.012405297305115, -0.0311314558844096,
-0.0757327062857556, -0.0438842167278324, 0.0534484315397847,
-0.00263093389834701, -0.0407968808622812, 0.0513899218257041,
0.161084849737891, 0.10414140315619, -0.105434010860225, 0.0317302108906947,
0.105366400083279, -0.256112774850456, -0.151872260620654, -0.0476317463908522
)), row.names = c(NA, -110L), class = c("tbl_df", "tbl", "data.frame"
))

If the problem is to find where the fit column starts rising then fit a curve made up of a horizontal line segment followed by a sloped line segment (red in the graph) and report the changepoint (Date0 and dashed line in graph).
# calculate starting values, st
fm0 <- lm(fit ~ Date, data, subset = seq(to = nrow(data), length = 20))
st <- c(mean(data$fit[1:20]), coef(fm0))
names(st) <- c("a0", "a", "b")
fm <- nls(fit ~ pmax(a0, a + b * as.numeric(Date)), data, start = st)
# solve a0 = a + b * Date0 for Date0 using calculated a0, a and b
Date0 <- with(as.list(coef(fm)), .POSIXct((a0 - a)/b))
plot(fit ~ Date, data, ylab = "")
lines(fitted(fm) ~ Date, data, col = "red")
abline(v = Date0, lty = 2)
Date0
## [1] "2017-03-21 07:53:56 EDT"

How to change period to 1 minute in Bloomberg API for R

EDIT:
updated the code but still have not been able to get intraday, 1 minute data.
I have the following code:
suppressMessages(library("Rblpapi"))
conn = blpConnect()
sec<-c("eurusd curncy", "eur2y curncy", "eur12m curncy")
flds<-"PX_LAST"
ovr = c("PERIOD" = "1")
a = as.data.frame(bdh(sec,flds,start.date=as.Date("2019-01-01 07:00"),end.date=as.Date("2019-01-31 07:00"), include.non.trading.days="FALSE", overrides = ovr))
Can't seem to get the period to change to 1 minute intervals.

netcdf - CDO monmean

I have a netcdf file, with a daily time step, that I wish to convert to a monthly time step.
The time is formatted as follow:
double time(time) ;
time:standard_name = "time" ;
time:long_name = "time" ;
time:bounds = "time_bnds" ;
time:units = "days since 2000-01-01" ;
time:calendar = "standard" ;
time:axis = "T" ;
When I convert to monthly time step using the command:
cdo monmean input.nc output.nc
Everything works fine except that the time output is strange:
time = "2000-01-16", "2000-02-15", "2000-03-16", "2000-04-15 12",
"2000-05-16", "2000-06-15 12", "2000-07-16", "2000-08-16",
"2000-09-15 12", "2000-10-16", "2000-11-15 12", "2000-12-16";
I wish to replace the day on the monthly value by the first day of the month and also remove those odd 12's for the time that appear. The desired output:
time = "2000-01-01", "2000-02-01", "2000-03-01", "2000-04-01",
"2000-05-01", "2000-06-01", "2000-07-01", "2000-08-01",
"2000-09-01", "2000-10-01", "2000-11-01", "2000-12-01";
Any hints is appreciate

cdo --timestat_date first monmean input.nc output.nc
works for me, I hope it's helpful! It places the timestamp at the first step of the averaging period, whereas the default is in the middle. (There is also a --timestat_date last if one want to do the opposite and put it at the last step of the window)

Rollapply for rolling regression with different window sizes, save coefficients in matrixes

I'm trying to perform a rolling linear regression, whit overlapping windows and for different window sizes. I want to save the result in two tables, one with window size and slopes and one wiht window size a and intercepts.
My data set is now saved as a xts object. I'm trying to make a for-loop to generate the different window-sizes and then do a rolling regression between the two variables, with a overlapping window using rollapply. Then I would like that the coefficients for every regression get saved as a row in a matrix, that can be built on, so that every window size corresponds to one row. I have looked at a lot of previous questions, but I can not get it right.
I have hourly data for 2 variables, this is the data for the x-variable:
> dput(Xq$x)
structure(c(339.76625, 176.7196875, 142.8063125, 118.5785625,
102.0514375, 86.01156251, 86.99050001, 85.0089375, 380.0010626,
2114.279375, 2157.76875, 2442.575, 2562.6375, 2537.7125, 2581.888125,
1947.7575, 834.7918751, 765.1525001, 989.698125, 1433.2, 429.9081251,
388.333125, 1500.995, 1523.816625, 1529.090126, 439.746875, 1029.03625,
1470.07825, 1454.844813, 1448.8455, 413.7550001, 334.03125, 566.7362501,
1609.184375, 1509.266876, 1403.1425, 1142.695625, 795.138125,
1562.881875, 933.190625, 549.085625, 500.135, 535.83, 444.3493751,
356.386875, 268.1875001, 141.1878125, 119.658875, 162.0425625,
153.6944375, 161.0025, 151.37025, 155.9567501, 123.64825, 145.664125,
141.4480625, 232.2009375, 913.6480001, 2227.08125, 1870.618751,
2190.606251, 1357.81625, 824.555625, 699.99, 703.695, 620.74625,
581.2375001, 393.391875, 349.96625, 188.5445625, 154.03125, 142.0825,
339.7915625, 139.8941875, 154.6279376, 171.6016875, 151.1387501,
159.7665, 168.62025, 180.7596875, 447.2746875, 1550.11625, 1887.275,
1446.883125, 699.590625, 983.9925, 1319.62875, 769.505625, 812.713125,
638.1437501, 585.8312501, 626.7875, 1924.293125, 629.855625,
349.88375, 1038.54875, 1588.82225, 548.3475, 1053.368125, 964.825625,
990.324375, 1327.160625, 395.5862501, 421.8154376, 1308.551875,
1334.045, 1633.408125, 886.2662501, 1618.7975, 1325.59625, 847.0693751,
667.501875, 594.0531251, 249.920625, 210.8869376, 224.3725001,
209.6325, 213.64875, 210.3175001, 194.54, 90.1085625, 71.12575001,
70.69412501, 69.6075, 68.404625, 67.88837501, 67.317125, 67.220375,
67.696625, 273.2732501, 606.475625, 1277.3425, 913.24875, 612.5462501,
571.42125, 503.27375, 316.68375, 208.7436251, 127.283, 126.12375,
102.504625, 78.7938125, 70.0183125, 69.97900001, 77.7586875,
76.60543751, 94.5235625, 103.0095625, 105.1938125, 103.8090625,
104.005625, 104.544875, 105.2036875, 104.6611875, 375.7004375,
793.7725001, 782.544375, 257.065625, 387.1487501, 571.1912501,
356.6406251, 243.404375, 220.698125, 182.5680626, 161.060625,
166.5178125, 137.9805, 129.6253125, 99.4291875, 75.88075001,
65.769875, 62.62937501, 59.697875, 57.68675, 56.67275, 56.69425001,
57.296, 638.95125, 1210.92875, 990.09, 907.895625, 1122.704375,
569.139375, 295.6725, 263.21, 236.3885625, 153.935, 148.0899375,
161.72675, 102.1535, 115.8895, 114.2579375, 95.0023125, 90.72425,
84.2715, 71.06631251, 63.39812501, 75.2416875, 73.9420625, 72.26237501,
402.8870625, 1148.9775, 1058.86875, 896.47875, 890.9687501, 833.645625,
670.310625, 298.576875, 306.538125, 386.403125, 196.8343751,
174.0815, 156.6158126, 117.9345625, 105.095, 90.24362501, 87.08800001,
86.1628125, 105.3179375, 107.312625, 114.089, 104.85825, 95.9281875,
126.2874375, 529.7325626, 1533.605625, 1089.700625, 760.47, 1905.04125,
1744.460625, 1346.120625, 847.90625, 459.4625, 472.11, 474.5068751,
481.83875, 937.3618751, 267.1339376, 137.7895625, 124.0425625,
188.9461875, 108.8915, 111.67825, 84.319125, 127.1240625, 117.519125,
104.16225, 105.2490625, 158.5540625, 1215.278063, 1275.396875,
1301.14375, 1108.9225, 1280.390625, 1434.5225, 1038.926875, 599.3218751,
665.22875, 646.935, 487.604375, 236.8148126, 300.6626876, 830.2973126,
1286.612313, 403.0063751, 405.9081875, 156.5465625, 101.7661875,
100.2301875, 144.424625, 126.781125, 143.940625, 365.5354376,
849.313125, 1729.3175, 1762.6625, 1733.968751, 1275.4375, 1332.930625,
620.5625, 504.115625, 538.4556251, 433.5725, 337.094375, 225.183125,
184.7196251, 156.7545, 130.23325, 134.0796875, 145.196125, 153.243,
141.096375, 414.8042501, 304.7252501, 204.1889375, 129.7909375,
555.0276251, 1734.235625, 1780.209375, 1462.7825, 967.4743751,
1248.791875, 1001.655625, 662.92875, 546.801875, 393.72375, 308.85125,
259.985625, 280.439375, 274.1312501, 310.0375001, 270.3, 201.9288125,
219.6061875, 208.372625, 156.2023125, 162.5355626, 167.823875,
151.3928126, 151.42875, 293.7148125, 434.9101875, 943.9243751,
995.955, 614.055, 1387.864375, 1752.330626, 1238.1725, 1260.93,
565.383125, 253.179375, 275.66625, 280.4912501, 222.684375, 338.079875,
201.9365625, 136.7436875, 146.5479375, 142.794625, 150.3969375,
134.6478125, 88.22068751, 70.6445, 71.73737501, 243.176, 1559.0925,
1527.626876, 1260.648125, 1251.564375, 714.1075001, 1124.58,
901.6018751, 551.9075, 303.1025, 355.435625, 372.32375, 235.62125,
185.39475, 161.7701876, 155.4823751), index = structure(c(1438380000,
1438383600, 1438387200, 1438390800, 1438394400, 1438398000, 1438401600,
1438405200, 1438408800, 1438412400, 1438416000, 1438419600, 1438423200,
1438426800, 1438430400, 1438434000, 1438437600, 1438441200, 1438444800,
1438448400, 1438452000, 1438455600, 1438459200, 1438462800, 1438466400,
1438470000, 1438473600, 1438477200, 1438480800, 1438484400, 1438488000,
1438491600, 1438495200, 1438498800, 1438502400, 1438506000, 1438509600,
1438513200, 1438516800, 1438520400, 1438524000, 1438527600, 1438531200,
1438534800, 1438538400, 1438542000, 1438545600, 1438549200, 1438552800,
1438556400, 1438560000, 1438563600, 1438567200, 1438570800, 1438574400,
1438578000, 1438581600, 1438585200, 1438588800, 1438592400, 1438596000,
1438599600, 1438603200, 1438606800, 1438610400, 1438614000, 1438617600,
1438621200, 1438624800, 1438628400, 1438632000, 1438635600, 1438639200,
1438642800, 1438646400, 1438650000, 1438653600, 1438657200, 1438660800,
1438664400, 1438668000, 1438671600, 1438675200, 1438678800, 1438682400,
1438686000, 1438689600, 1438693200, 1438696800, 1438700400, 1438704000,
1438707600, 1438711200, 1438714800, 1438718400, 1438722000, 1438725600,
1438729200, 1438732800, 1438736400, 1438740000, 1438743600, 1438747200,
1438750800, 1438754400, 1438758000, 1438761600, 1438765200, 1438768800,
1438772400, 1438776000, 1438779600, 1438783200, 1438786800, 1438790400,
1438794000, 1438797600, 1438801200, 1438804800, 1438808400, 1438812000,
1438815600, 1438819200, 1438822800, 1438826400, 1438830000, 1438833600,
1438837200, 1438840800, 1438844400, 1438848000, 1438851600, 1438855200,
1438858800, 1438862400, 1438866000, 1438869600, 1438873200, 1438876800,
1438880400, 1438884000, 1438887600, 1438891200, 1438894800, 1438898400,
1438902000, 1438905600, 1438909200, 1438912800, 1438916400, 1438920000,
1438923600, 1438927200, 1438930800, 1438934400, 1438938000, 1438941600,
1438945200, 1438948800, 1438952400, 1438956000, 1438959600, 1438963200,
1438966800, 1438970400, 1438974000, 1438977600, 1438981200, 1438984800,
1438988400, 1438992000, 1438995600, 1438999200, 1439002800, 1439006400,
1439010000, 1439013600, 1439017200, 1439020800, 1439024400, 1439028000,
1439031600, 1439035200, 1439038800, 1439042400, 1439046000, 1439049600,
1439053200, 1439056800, 1439060400, 1439064000, 1439067600, 1439071200,
1439074800, 1439078400, 1439082000, 1439085600, 1439089200, 1439092800,
1439096400, 1439100000, 1439103600, 1439107200, 1439110800, 1439114400,
1439118000, 1439121600, 1439125200, 1439128800, 1439132400, 1439136000,
1439139600, 1439143200, 1439146800, 1439150400, 1439154000, 1439157600,
1439161200, 1439164800, 1439168400, 1439172000, 1439175600, 1439179200,
1439182800, 1439186400, 1439190000, 1439193600, 1439197200, 1439200800,
1439204400, 1439208000, 1439211600, 1439215200, 1439218800, 1439222400,
1439226000, 1439229600, 1439233200, 1439236800, 1439240400, 1439244000,
1439247600, 1439251200, 1439254800, 1439258400, 1439262000, 1439265600,
1439269200, 1439272800, 1439276400, 1439280000, 1439283600, 1439287200,
1439290800, 1439294400, 1439298000, 1439301600, 1439305200, 1439308800,
1439312400, 1439316000, 1439319600, 1439323200, 1439326800, 1439330400,
1439334000, 1439337600, 1439341200, 1439344800, 1439348400, 1439352000,
1439355600, 1439359200, 1439362800, 1439366400, 1439370000, 1439373600,
1439377200, 1439380800, 1439384400, 1439388000, 1439391600, 1439395200,
1439398800, 1439402400, 1439406000, 1439409600, 1439413200, 1439416800,
1439420400, 1439424000, 1439427600, 1439431200, 1439434800, 1439438400,
1439442000, 1439445600, 1439449200, 1439452800, 1439456400, 1439460000,
1439463600, 1439467200, 1439470800, 1439474400, 1439478000, 1439481600,
1439485200, 1439488800, 1439492400, 1439496000, 1439499600, 1439503200,
1439506800, 1439510400, 1439514000, 1439517600, 1439521200, 1439524800,
1439528400, 1439532000, 1439535600, 1439539200, 1439542800, 1439546400,
1439550000, 1439553600, 1439557200, 1439560800, 1439564400, 1439568000,
1439571600, 1439575200, 1439578800, 1439582400, 1439586000, 1439589600,
1439593200, 1439596800, 1439600400, 1439604000, 1439607600, 1439611200,
1439614800, 1439618400, 1439622000, 1439625600, 1439629200, 1439632800,
1439636400, 1439640000, 1439643600, 1439647200, 1439650800, 1439654400,
1439658000, 1439661600, 1439665200, 1439668800, 1439672400), tzone = "UTC", tclass = c("chron",
"dates", "times")), class = c("xts", "zoo"), .indexCLASS = c("chron",
"dates", "times"), tclass = c("chron", "dates", "times"), .indexTZ = "UTC", tzone = "UTC", .Dim = c(360L,
1L), .Dimnames = list(NULL, "x"))
And my y-variable:
> dput(Xq$y)
structure(c(-0.274050833, -0.236638333, -0.1994325, -0.174091667,
-0.153273333, -0.136978333, -0.124748333, -0.117348333, -0.147061667,
-0.346170833, -0.517939167, -0.575585833, -0.595914167, -0.563639167,
-0.53403, -0.5500525, -0.505650833, -0.471713333, -0.472485833,
-0.547393333, -0.390174167, -0.321545, -0.29781, -0.26912, -0.345084167,
-0.367618333, -0.279933333, -0.256805, -0.2514675, -0.314349167,
-0.3594375, -0.33482, -0.369094167, -0.4801075, -0.554780833,
-0.600498333, -0.604796667, -0.544491667, -0.636653333, -0.568401667,
-0.485494167, -0.453199167, -0.417475, -0.38417, -0.341585833,
-0.2821625, -0.248325, -0.2230575, -0.2449075, -0.2385375, -0.231885,
-0.214125, -0.194190833, -0.178575833, -0.1677675, -0.1615725,
-0.1739675, -0.2432125, -0.402414167, -0.448185833, -0.563599167,
-0.5855025, -0.55586, -0.516350833, -0.47892, -0.4603325, -0.434146667,
-0.37602, -0.320976667, -0.267863333, -0.250915, -0.241764167,
-0.271475, -0.2170225, -0.206605, -0.2088, -0.214511667, -0.2086825,
-0.203060833, -0.192895, -0.212629167, -0.319143333, -0.42647,
-0.4623275, -0.467844167, -0.506395833, -0.507088333, -0.496953333,
-0.511156667, -0.487846667, -0.4455525, -0.398383333, -0.565926667,
-0.429720833, -0.314555, -0.418586667, -0.578256667, -0.414858333,
-0.345411667, -0.3088925, -0.304373333, -0.334221667, -0.305029167,
-0.273269167, -0.315901667, -0.409731667, -0.500245833, -0.505959167,
-0.54742, -0.574725, -0.548458333, -0.5560675, -0.50246, -0.411618333,
-0.35965, -0.331884167, -0.312573333, -0.298478333, -0.289144167,
-0.274429167, -0.2218225, -0.198025833, -0.191955833, -0.1780825,
-0.157910833, -0.135935, -0.116965833, -0.099886667, -0.0864975,
-0.188904167, -0.325656667, -0.444044167, -0.5050425, -0.54236,
-0.547500833, -0.5399775, -0.521490833, -0.444911667, -0.388053333,
-0.327650833, -0.2478, -0.2026775, -0.1693425, -0.140848333,
-0.130440833, -0.103501667, -0.088843333, -0.09344, -0.126231667,
-0.158463333, -0.181145, -0.185095, -0.11657, -0.0349225, -0.223260833,
-0.431800833, -0.516, -0.487566667, -0.49941, -0.541773333, -0.511953333,
-0.4404425, -0.4021125, -0.380635, -0.3301275, -0.302468333,
-0.311290833, -0.2774125, -0.223738333, -0.192925, -0.176034167,
-0.158151667, -0.136234167, -0.117050833, -0.101201667, -0.08872,
-0.0855675, -0.257761667, -0.424794167, -0.528980833, -0.56834,
-0.591263333, -0.56574, -0.53178, -0.5410825, -0.552130833, -0.501294167,
-0.44576, -0.402855833, -0.316163333, -0.2515275, -0.213383333,
-0.186723333, -0.174315833, -0.170100833, -0.154446667, -0.147075833,
-0.169245, -0.188474167, -0.187344167, -0.2586975, -0.440815,
-0.534344167, -0.598391667, -0.613878333, -0.624658333, -0.583091667,
-0.508740833, -0.518310833, -0.51102, -0.4203825, -0.364895833,
-0.31302, -0.27689, -0.254130833, -0.232273333, -0.218198333,
-0.22501, -0.235655833, -0.242728333, -0.260448333, -0.263113333,
-0.243530833, -0.222301667, -0.274098333, -0.3974225, -0.484443333,
-0.506165, -0.6136375, -0.631805, -0.596274167, -0.539795, -0.446769167,
-0.398489167, -0.3986925, -0.4098625, -0.484515833, -0.375961667,
-0.291685, -0.273963333, -0.2621025, -0.2054525, -0.177625833,
-0.1564625, -0.141883333, -0.130685833, -0.121943333, -0.112834167,
-0.136975, -0.3204725, -0.458004167, -0.5262175, -0.527530833,
-0.520574167, -0.5800325, -0.561841667, -0.4909225, -0.4641625,
-0.443126667, -0.405705833, -0.3098625, -0.286379167, -0.30477,
-0.395341667, -0.38505, -0.356219167, -0.263050833, -0.219625833,
-0.197383333, -0.2019775, -0.226838333, -0.2333075, -0.2890475,
-0.3341175, -0.43458, -0.520441667, -0.577875, -0.568123333,
-0.551936667, -0.463691667, -0.468790833, -0.4747725, -0.4125925,
-0.394731667, -0.380213333, -0.308688333, -0.279549167, -0.249766667,
-0.233964167, -0.229904167, -0.244835833, -0.232436667, -0.215466667,
-0.198559167, -0.184533333, -0.185376667, -0.246823333, -0.395918333,
-0.4956775, -0.540474167, -0.5402375, -0.577863333, -0.561466667,
-0.503130833, -0.455221667, -0.4401875, -0.4187675, -0.389215,
-0.345275, -0.3378175, -0.348759167, -0.325149167, -0.29995,
-0.289409167, -0.291635, -0.301183333, -0.274799167, -0.2443375,
-0.2254225, -0.21272, -0.245265, -0.295081667, -0.356848333,
-0.4258325, -0.4329175, -0.487074167, -0.595525, -0.59333, -0.564645,
-0.464548333, -0.4294775, -0.425609167, -0.403769167, -0.353311667,
-0.282888333, -0.2464575, -0.231771667, -0.2275425, -0.224230833,
-0.218419167, -0.1902275, -0.161319167, -0.143495833, -0.133691667,
-0.181291667, -0.36056, -0.46681, -0.5194575, -0.532989167, -0.4899375,
-0.533224167, -0.4976575, -0.428966667, -0.412929167, -0.416463333,
-0.366666667, -0.316356667, -0.3023825, -0.282655, -0.267275833
), index = structure(c(1438380000, 1438383600, 1438387200, 1438390800,
1438394400, 1438398000, 1438401600, 1438405200, 1438408800, 1438412400,
1438416000, 1438419600, 1438423200, 1438426800, 1438430400, 1438434000,
1438437600, 1438441200, 1438444800, 1438448400, 1438452000, 1438455600,
1438459200, 1438462800, 1438466400, 1438470000, 1438473600, 1438477200,
1438480800, 1438484400, 1438488000, 1438491600, 1438495200, 1438498800,
1438502400, 1438506000, 1438509600, 1438513200, 1438516800, 1438520400,
1438524000, 1438527600, 1438531200, 1438534800, 1438538400, 1438542000,
1438545600, 1438549200, 1438552800, 1438556400, 1438560000, 1438563600,
1438567200, 1438570800, 1438574400, 1438578000, 1438581600, 1438585200,
1438588800, 1438592400, 1438596000, 1438599600, 1438603200, 1438606800,
1438610400, 1438614000, 1438617600, 1438621200, 1438624800, 1438628400,
1438632000, 1438635600, 1438639200, 1438642800, 1438646400, 1438650000,
1438653600, 1438657200, 1438660800, 1438664400, 1438668000, 1438671600,
1438675200, 1438678800, 1438682400, 1438686000, 1438689600, 1438693200,
1438696800, 1438700400, 1438704000, 1438707600, 1438711200, 1438714800,
1438718400, 1438722000, 1438725600, 1438729200, 1438732800, 1438736400,
1438740000, 1438743600, 1438747200, 1438750800, 1438754400, 1438758000,
1438761600, 1438765200, 1438768800, 1438772400, 1438776000, 1438779600,
1438783200, 1438786800, 1438790400, 1438794000, 1438797600, 1438801200,
1438804800, 1438808400, 1438812000, 1438815600, 1438819200, 1438822800,
1438826400, 1438830000, 1438833600, 1438837200, 1438840800, 1438844400,
1438848000, 1438851600, 1438855200, 1438858800, 1438862400, 1438866000,
1438869600, 1438873200, 1438876800, 1438880400, 1438884000, 1438887600,
1438891200, 1438894800, 1438898400, 1438902000, 1438905600, 1438909200,
1438912800, 1438916400, 1438920000, 1438923600, 1438927200, 1438930800,
1438934400, 1438938000, 1438941600, 1438945200, 1438948800, 1438952400,
1438956000, 1438959600, 1438963200, 1438966800, 1438970400, 1438974000,
1438977600, 1438981200, 1438984800, 1438988400, 1438992000, 1438995600,
1438999200, 1439002800, 1439006400, 1439010000, 1439013600, 1439017200,
1439020800, 1439024400, 1439028000, 1439031600, 1439035200, 1439038800,
1439042400, 1439046000, 1439049600, 1439053200, 1439056800, 1439060400,
1439064000, 1439067600, 1439071200, 1439074800, 1439078400, 1439082000,
1439085600, 1439089200, 1439092800, 1439096400, 1439100000, 1439103600,
1439107200, 1439110800, 1439114400, 1439118000, 1439121600, 1439125200,
1439128800, 1439132400, 1439136000, 1439139600, 1439143200, 1439146800,
1439150400, 1439154000, 1439157600, 1439161200, 1439164800, 1439168400,
1439172000, 1439175600, 1439179200, 1439182800, 1439186400, 1439190000,
1439193600, 1439197200, 1439200800, 1439204400, 1439208000, 1439211600,
1439215200, 1439218800, 1439222400, 1439226000, 1439229600, 1439233200,
1439236800, 1439240400, 1439244000, 1439247600, 1439251200, 1439254800,
1439258400, 1439262000, 1439265600, 1439269200, 1439272800, 1439276400,
1439280000, 1439283600, 1439287200, 1439290800, 1439294400, 1439298000,
1439301600, 1439305200, 1439308800, 1439312400, 1439316000, 1439319600,
1439323200, 1439326800, 1439330400, 1439334000, 1439337600, 1439341200,
1439344800, 1439348400, 1439352000, 1439355600, 1439359200, 1439362800,
1439366400, 1439370000, 1439373600, 1439377200, 1439380800, 1439384400,
1439388000, 1439391600, 1439395200, 1439398800, 1439402400, 1439406000,
1439409600, 1439413200, 1439416800, 1439420400, 1439424000, 1439427600,
1439431200, 1439434800, 1439438400, 1439442000, 1439445600, 1439449200,
1439452800, 1439456400, 1439460000, 1439463600, 1439467200, 1439470800,
1439474400, 1439478000, 1439481600, 1439485200, 1439488800, 1439492400,
1439496000, 1439499600, 1439503200, 1439506800, 1439510400, 1439514000,
1439517600, 1439521200, 1439524800, 1439528400, 1439532000, 1439535600,
1439539200, 1439542800, 1439546400, 1439550000, 1439553600, 1439557200,
1439560800, 1439564400, 1439568000, 1439571600, 1439575200, 1439578800,
1439582400, 1439586000, 1439589600, 1439593200, 1439596800, 1439600400,
1439604000, 1439607600, 1439611200, 1439614800, 1439618400, 1439622000,
1439625600, 1439629200, 1439632800, 1439636400, 1439640000, 1439643600,
1439647200, 1439650800, 1439654400, 1439658000, 1439661600, 1439665200,
1439668800, 1439672400), tzone = "UTC", tclass = c("chron", "dates",
"times")), class = c("xts", "zoo"), .indexCLASS = c("chron",
"dates", "times"), tclass = c("chron", "dates", "times"), .indexTZ = "UTC", tzone = "UTC", .Dim = c(360L,
1L), .Dimnames = list(NULL, "y"))
My code looks like this:
library(zoo)
mat_slope=matrix(nrow=i)
mat_intercept=matrix(nrow=i)
for(i in seq(from=24, to=240, by=24)){
mov.reg<- rollapplyr(Xq,
width= i,
by=24,
FUN = function(y,x)
{
coefficients(lm(formula=y~x, data = Xq))
},
by.column=FALSE)
mat_slope[i] <- coefficients(mov.reg)[2]
mat_intercept[i] <-coefficients(mov.reg)[1]
mat_intercept=rbind(i,mat_intercept)
mat_slope[n,i]=rbind(i, mat_slope)
}
I either get an empty matrix as result, or the following error message:
Error in merge.xts(res, xts(, idx, attr(data, "frequency"))) :
(list) object cannot be coerced to type 'double'
If anyone have some inputs or ideas it would be very appreciated! Thank's a lot!

mov.reg is not an lm objct, so it does not make sense to take coefficients of it. Also there is a reference to chron in the dput output so we need to load the chron package.
Xq is not specified but rather two separate xts objects are shown so we have assumed that those two are called x and y. Then create a single zoo object z from them. Now define a Coef function which takes a matrix with x and y columns and performs the indicated regression returning the coefficients. Also define roll which takes the width as input and runs rollapplyr with the desired arguments. Now use lapply to run roll with each width producing a list of zoo objects. This list has one component for each element in widths consisting of a zoo object with intercept and slope columns. Finally extract the intercepts from each component in the list and form a zoo object and do the same for the slopes. Note that if L is a list of zoo objects then do.call("merge", L) will produce a single zoo object from it.
library(xts) # this also loads zoo
library(chron)
# inputs are xts objects x and y
z <- cbind(x = as.zoo(x)[, 1], y = as.zoo(y)[, 1])
Coef <- function(m) coef(lm(y ~ x, as.data.frame(m)))
roll <- function(w) rollapplyr(z, w, Coef, by = 24, by.column = FALSE)
widths <- seq(24, 240, 24)
names(widths) <- widths # lapply will use these names for its output
L <- lapply(widths, roll)
intercepts <- do.call("merge", lapply(L, "[", TRUE, 1)) # extract 1st columns
slopes <- do.call("merge", lapply(L, "[", TRUE, 2)) # 2nd columns
Alternately do the rollapplyr twice replacing the last 3 lines with:
intercepts <- do.call("merge", lapply(widths, function(w) roll(w)[, 1]))
slopes <- do.call("merge", lapply(widths, function(w) roll(w)[, 2]))

Count the amount of observations in predetermined timestep

I have a large dataset of over 75.000 observations. Of these observations I have a list of date and time combinations. I want to calculate the observation frequency in a predetermined timestep (15, 30 or 60 minutes). The study period is from 2014-10-21 00:00 to 2015-10-21 23:59.
The raw data is stored in a DF, but date (as POSIXlt) and time (as character) are in different columns, so I combine them back into one column to create a POSIXct timestamp.
receiver$date2 = as.POSIXct(paste(receiver$date, receiver$time), format="%Y-
%m-%d %H:%M:%S")
dateseq = receiver$date2
dateseq is now (only a small fragment using dput()):
dateseq = structure(c(1414140420, 1414140720, 1414140960, 1414141080, 1414143540, 1414144980, 1414145940, 1414147380, 1414147440, 1414148100, 1414148280, 1414152720, 1414153740, 1414154520, 1414154580, 1414158540, 1414159380, 1414159680, 1414164240, 1414164300, 1414164840, 1414164900, 1414165500, 1414166100, 1414166220, 1414166460, 1414166520, 1414166820, 1414166880, 1414166940, 1414167300, 1414167360, 1414167480, 1414167780, 1414168380, 1414168440, 1414168800, 1414168860, 1414202040, 1414202220, 1414202280, 1414202700, 1414202820, 1414202880, 1414203660, 1414203960, 1414215180, 1414215300, 1414215900, 1414216560, 1414216860, 1414217220, 1414217280, 1414217460, 1414217580, 1414217700, 1414217820, 1414217880, 1414218240, 1414218720, 1414219380, 1414219800, 1414219920, 1414219980, 1414220160, 1414220280, 1414220820, 1414220880, 1414221000, 1414221960, 1414222080, 1414222200, 1414222320, 1414222500, 1414222560, 1414222860, 1414223640, 1414224780, 1414225800, 1414225920, 1414225980, 1414226040, 1414226100, 1414226220, 1414227240, 1414227420, 1414227600, 1414230300, 1414230540, 1414230840, 1414231140, 1414231320, 1414231440, 1414231560, 1414231800, 1414231860, 1414232040, 1414232160, 1414232400, 1414232520, 1414232640, 1414232700, 1414232760, 1414232880, 1414232940, 1414233060, 1414233180, 1414233240, 1414233300, 1414233420, 1414233480, 1414233660, 1414233720, 1414233780, 1414233840, 1414233960, 1414234080, 1414234320, 1414234440, 1414234560, 1414234620, 1414234740, 1414234860, 1414234980, 1414235040, 1414235280, 1414236240, 1414236300, 1414236420, 1414236540, 1414236840, 1414236900, 1414236960, 1414237020, 1414237260, 1414237560, 1414237860, 1414238280, 1414238400, 1414238460, 1414238580, 1414238640, 1414239180, 1414239300, 1414239360, 1414239480, 1414239540, 1414240440, 1414240860, 1414240920, 1414240980, 1414241040, 1414242000, 1414242180, 1414242480, 1414242540, 1414242660, 1414242720, 1414242840, 1414242900, 1414243800, 1414243920, 1414244280, 1414244460, 1414245240, 1414245600, 1414245660, 1414246080, 1414246500, 1414246680, 1414246740, 1414246920, 1414247340, 1414248180, 1414249320, 1414249560, 1414249860, 1414250340, 1414250520, 1414250640, 1414250760, 1414250880, 1414250940, 1414251060, 1414251240, 1414251900, 1414252020, 1414252080, 1414252200, 1414252260, 1414252380, 1414252440, 1414252440, 1414252500, 1414252560, 1414252680, 1414252980, 1414253160, 1414253460, 1414253580), class = c("POSIXct", "POSIXt"), tzone = "")
Then I want to have a timeseq that runs for the whole period (so also the days that don't have any observations) divided by the predetermined timestep.
timestep = 1800 # 1800 sec = 30 min
start = "2014-10-21 00:00"
end = "2015-10-21 23:59"
receiver = R125926
timeseq = seq(from = as.POSIXct(start), to = as.POSIXct(end), by = timestep)
Now I want to 'fill' a new dataframe with the timeseq in one column and the count data of how many observations (from dateseq) occurred in that time period.
EDIT
After some searching on the forum and adjusting some code, I came to one very simple method that brings me very close to what I want my results to look like:
det_interval = data.frame(table(cut(dateseq, breaks = "30 min")))
There's only two adjustments that I don't know how to do. Now it begins at the first record (e.g. when my first record is on 05.17 the interval it will use will be x.17 - x.47 (30min)), not at the start that I want (see the timeseq created above). So how can I make sure that this starts and ends at a predetermined date/time?

Resources