I hope you're doing well! I have a theoretical time-series analysis problem I hope that you can help me sort out.
To start, you'll find a reproducible example of my dataset below. Date is in a daily timescale. Q25 is 25th or lower quartile of my non-transformed data, Q75 is the 75th or upper quartile of my non-transformed data, fit is local weighted fit of the median, firstder is the first derivative of fit, and secondder is the second derivative of fit.
Plotting out fit produces two oscillations and then a steady increase in the data. Plotting the quartiles around that produces a large spread of data that narrows towards the increase in fit data. The first derivative shows the rate of change of the fit and this is where my issue comes in. I'm not sure where the increase in fit data starts based on the first derivative data. Logically, I know the signal-to-noise start date has to occur after March 7 (based on the quartiles), and before March 20 (before the steady increase in fit data). And this is also represented in the first derivative for about the same interval where the negative-to-positive inflection point changes on March 5th, becomes positive on March 16th, and then produces a stationary time series.
All that being said, should my exact start date be the change in the inflection point from the first derivative, or be the first positive value on March 16th?
I appreciate your time in this problem and any thoughts you may have!
data<-structure(list(Date = structure(c(1485950474, 1486036874, 1486123274,
1486209674, 1486296074, 1486382474, 1486468874, 1486555274, 1486641674,
1486728074, 1486814474, 1486900874, 1486987274, 1487073674, 1487160074,
1487246474, 1487332874, 1487419274, 1487505674, 1487592074, 1487678474,
1487764874, 1487851274, 1487937674, 1488024074, 1488110474, 1488196874,
1488283274, 1488369674, 1488456074, 1488542474, 1488628874, 1488715274,
1488801674, 1488888074, 1488974474, 1489060874, 1489147274, 1489233674,
1489320074, 1489406474, 1489492874, 1489579274, 1489665674, 1489752074,
1489838474, 1489924874, 1490011274, 1490097674, 1490184074, 1490270474,
1490356874, 1490443274, 1490529674, 1490616074, 1490702474, 1490788874,
1490875274, 1490961674, 1491048074, 1491134474, 1491220874, 1491307274,
1491393674, 1491480074, 1491566474, 1491652874, 1491739274, 1491825674,
1491912074, 1491998474, 1492084874, 1492171274, 1492257674, 1492344074,
1492430474, 1492516874, 1492603274, 1492689674, 1492776074, 1492862474,
1492948874, 1493035274, 1493121674, 1493208074, 1493294474, 1493380874,
1493467274, 1493553674, 1493640074, 1493726474, 1493812874, 1493899274,
1493985674, 1494072074, 1494158474, 1494244874, 1494331274, 1494417674,
1494504074, 1494590474, 1494676874, 1494763274, 1494849674, 1494936074,
1495022474, 1495108874, 1495195274, 1495281674, 1495368074), tzone = "UTC", class = c("POSIXct",
"POSIXt")), Q25 = c(-1.61495132528742, -3.86616056128065, -3.92140420424278,
-4.8011229557052, -8.64427034627082, -3.11323607034871, -4.3673083843457,
-1.45023104534208, 0.395769745934938, -1.49394189431791, -3.54063822876105,
-4.36090193633662, -0.966958995958447, -2.43233048854294, -0.181367797683111,
0.826258942687981, 3.36833418895383, -6.8991417494414, -1.15773470862185,
-1.75360705873163, 1.83790453304777, 2.11575746130393, -3.82025172988123,
0.679651741170909, -4.64628184041103, -6.91923314565111, 0.550274303541761,
0.104011128328036, -0.895257855280075, -0.801630235696042, 2.27958927430356,
2.98003963398985, 3.41649824319921, 1.56559818977215, -2.20923132476973,
0.552658760232765, 0.15158829140461, -4.75454688546242, -0.595460561248954,
-2.53729443345183, -0.826010503400985, -5.20578683534568, -2.78364193219594,
-3.62503323095109, 3.37820215582788, -2.53645164034493, -1.76051141957494,
-1.0256290530567, 1.94178279643985, 0.261239031590387, 0.00321585342072063,
2.87814873140354, -2.26732156613212, 2.65097224867168, -4.16746046231376,
1.64816233695592, 3.50505415841016, 2.83685877611882, 1.66353660199615,
2.27900517713667, 5.47721995923733, -5.31044894311933, 7.30753839733595,
5.50143585044911, -1.25129055380416, -2.41051058119916, 3.69266303212359,
2.28752278841533, -0.275687673398348, 5.74597173218469, 6.5773422259343,
3.72096844335478, 2.05388534852328, 5.41063696868948, 0.526467452167141,
1.60445671702256, -1.80394989627014, 1.56432488418924, 5.95370989889123,
7.94953250403525, 4.09121878799004, 2.11516919787794, -2.12808005361608,
6.77215849921842, 9.53718510298556, 4.16562173164636, 10.4573226478082,
7.703077796612, 7.55811710979136, 4.47194951592662, 10.2104312432178,
11.3454383477984, 0.997649090931488, 4.84898050707927, 10.8819209584302,
8.06296236341084, 11.3317616787558, 7.51878628894305, 7.87729934765305,
11.9108509727303, 6.77401202490232, 5.36297357453455, 10.6362047038983,
8.68979831512869, 4.0465996534104, 11.9579904470733, 9.41141176380086,
10.5754750604254, 12.6944336852953, 7.61563466861022), Q75 = c(5.93775779359077,
26.4536084846094, 7.92690107568623, 16.195405687679, 3.47567054091916,
34.9690262666155, 15.5126126583077, 24.4425589002446, 29.7425859431597,
23.1420118192775, 26.827758017105, 18.6306368759596, 19.759179203689,
10.0667740183259, 30.9080218485755, 10.0628623899296, 21.1120424008512,
12.1232187464341, 14.9571040303508, 11.4927011052638, 16.1617172813173,
19.0606972964125, 8.39991659547325, 9.5080530252195, 10.2717546026802,
12.018391863395, 27.2666992661895, 12.5172584337237, 19.9658806224003,
6.90019918091751, 18.4119063276997, 23.2991253786256, 27.95161418973,
16.9477966472485, 26.3880458021082, 19.2178725103802, 5.58699033890406,
9.82525729279156, 6.22139350667344, 5.6625294221828, 8.18283315939774,
4.78856479855966, 4.91215612536983, 5.35278870440784, 15.7471499356884,
7.95473965312171, 7.58463611165082, 6.03119890210746, 9.88624343762245,
6.66377352843609, 6.92675024060609, 7.20403099201013, 6.96877369392089,
17.7034248870798, 6.22890341708267, 6.1624397247754, 23.3856864094132,
7.13518162203812, 6.96344109315883, 7.69414570220079, 18.0859103957135,
7.52300478408242, 10.1635801549871, 10.021556657451, 8.51746254314866,
7.83000625461296, 15.4938419153615, 8.6844260972191, 8.07596479745038,
13.1423674521087, 8.04161364299224, 10.7442773622841, 8.58410892324644,
9.08436532340561, 8.84748510783176, 9.27529549461203, 9.01978932806698,
9.99776533859531, 9.61123990151036, 11.2228855544025, 10.3285714984086,
10.7107229417799, 11.452541129334, 11.9951421202043, 11.3568792509498,
11.139621487692, 12.957244784325, 13.1010906952192, 11.8445972599726,
12.8124554609003, 12.1817389611984, 13.4529860098547, 13.1808997426024,
12.568956945967, 13.9405958892683, 14.4445923505263, 14.5816203429081,
12.798362023978, 13.7926596005317, 14.3284196983115, 14.3967490595795,
14.3699332949429, 13.8061418130819, 15.4045229902535, 15.328632395916,
15.5928587109464, 15.5111381098579, 15.7167488979248, 16.4121827249844,
16.7700564366026), fit = c(1.3157822724014, 1.44491806546299,
1.67963756121542, 1.96834398237369, 2.32222986513481, 2.73223146146706,
3.16143742264514, 3.74278329406317, 4.4673163398484, 5.08529278937518,
5.58735598987316, 6.01592790788482, 6.19893270175371, 6.0219082198616,
5.64253432163072, 5.29694818196536, 4.89670493804841, 4.35145910275626,
3.89449691453349, 3.48150649031492, 3.06858491643756, 2.88963188544926,
3.13399806321574, 3.62311989322663, 4.03902573446563, 4.40598627768245,
4.84291047423098, 5.1737840740012, 5.3972440468493, 5.5747020603732,
5.62430591107552, 5.42843052467024, 5.07513358262307, 4.79108701506415,
4.59907825712695, 4.39731440509327, 4.22559688081583, 4.10100609028878,
4.00444369172723, 3.92144298531529, 3.82259220819525, 3.72499526558926,
3.68395895980124, 3.69588308031619, 3.73924432798967, 3.84246487218137,
4.07884774763199, 4.41108295888359, 4.70167312999791, 4.95537881350854,
5.2206483181831, 5.42551590243433, 5.52148736399275, 5.55736071284688,
5.60710852579646, 5.65757759073701, 5.68911425674423, 5.76594044238814,
5.93786454015275, 6.15175825295678, 6.31743846502224, 6.40077523837882,
6.45704948591979, 6.53019436816257, 6.59356685208809, 6.63353784524384,
6.71356141899707, 6.88849022040772, 7.11437487009308, 7.30646639975639,
7.43724432723552, 7.55279324817994, 7.67877181101032, 7.76924002146674,
7.83161170884946, 7.97157625691941, 8.25223488219952, 8.60947602940562,
8.95816992458796, 9.34076728750423, 9.77554331222275, 10.1411049362597,
10.3842988541376, 10.5696053585185, 10.7520817841281, 10.9357595672387,
11.0970528791622, 11.2495931571849, 11.3764752236255, 11.4864715266717,
11.6317299424136, 11.8381584436134, 12.0667779318613, 12.2724056764894,
12.462010561811, 12.6517333832877, 12.8101492769744, 12.9055352762602,
12.9678598772259, 13.0582354099638, 13.1489397497677, 13.2204738414797,
13.346284619515, 13.6054940294766, 13.9436193637562, 14.2337005769519,
14.5449448398809, 14.8895799498019, 15.0551768009747, 15.0689572800127
), firstder = c(0.0542499277820437, 0.193160412687084, 0.264645386746196,
0.318230646770668, 0.390583391620104, 0.410606699200811, 0.484714112557398,
0.683182658658343, 0.699350916534123, 0.546311900646561, 0.476582322984034,
0.33921923563074, 0.000346679118119919, -0.32275830659655, -0.377372654859586,
-0.342379980870621, -0.492111485610006, -0.524917784293232, -0.414059192641829,
-0.430018688265099, -0.343482693656914, 0.0295127267198723, 0.42189373253822,
0.482044173095213, 0.364522990904745, 0.40991488301477, 0.40715895907959,
0.264020778627613, 0.200548459021332, 0.136695124879259, -0.0667758528503706,
-0.308783766357995, -0.344835056787729, -0.22338628389576, -0.19056674389956,
-0.195775242472453, -0.146360055189657, -0.107867992742261, -0.0856184200473131,
-0.0883963049921002, -0.106496806989568, -0.0747428483921662,
-0.0103234284849929, 0.028493059030597, 0.0620691939868203, 0.163240621281308,
0.304123951137378, 0.325609827601989, 0.261609166722046, 0.261432729205552,
0.249586474110962, 0.150199026157553, 0.0521536950613295, 0.0370628072573624,
0.0565243651980056, 0.0371337817771211, 0.0409727028064402, 0.124422569131023,
0.207609809433488, 0.20232516927351, 0.121600832063498, 0.058433044321534,
0.0638003776220697, 0.0745713396178918, 0.0471802520722933, 0.0467708263829785,
0.126045851395065, 0.213953247074989, 0.220308792495525, 0.158550331399022,
0.11422743390592, 0.123806714974779, 0.114997378074604, 0.0651990840907102,
0.0828996021118185, 0.210617558388392, 0.336478451788591, 0.356675237198802,
0.354610868118913, 0.419333862640583, 0.419974858146042, 0.301270480481834,
0.201419853206041, 0.17882844049566, 0.186628379656891, 0.172934534594114,
0.156583940148236, 0.142289490196014, 0.11234075824169, 0.119081439314575,
0.177295034391252, 0.226764155293772, 0.22057696671022, 0.193643700730051,
0.190744241252391, 0.181381161962744, 0.127949080877661, 0.0681406193708671,
0.0729227267768433, 0.0983314755975622, 0.0766175682172481, 0.0819140886989596,
0.188151474480757, 0.320764600927798, 0.32011829707578, 0.283266397091015,
0.351814702578002, 0.276441515194414, 0.0724489974588587, -0.0273030060468944
), secondder = c(0.172623240328004, 0.105197729482076, 0.0377722186361492,
0.0693983014127931, 0.0753071882860794, -0.0352605731246656,
0.18347539983784, 0.213461692364051, -0.181125176612492, -0.124952855162631,
-0.0145063001624228, -0.260219874544165, -0.417525238481075,
-0.228684732948264, 0.119456036422192, -0.0494706884442619, -0.249992321034509,
0.184379723668058, 0.037337459634748, -0.0692564508812885, 0.242328440097658,
0.503662400655915, 0.281099610980781, -0.160798729866795, -0.0742436345141417,
0.165027418734192, -0.170539266604553, -0.1157370942994, -0.0112075449131641,
-0.116499123370982, -0.290442832088276, -0.193572994926973, 0.121470414067507,
0.12142713171643, -0.055788051724031, 0.0453710545782453, 0.0534593199873461,
0.0235248049074466, 0.0209743404824492, -0.0265301103720232,
-0.00967089362291196, 0.0731788108177152, 0.0556600289966314,
0.0219729460345484, 0.0451793238778984, 0.157163530711077, 0.124603129001063,
-0.081631376071841, -0.0463699456880455, 0.0460170706550578,
-0.0697095808442372, -0.129065315062581, -0.0670253471298663,
0.0368435715219322, 0.0020795443593542, -0.0408607112011232,
0.0485385532597613, 0.118361179389404, 0.0480133012155273, -0.058582581535485,
-0.102866092884539, -0.0234694825993884, 0.0342041492004599,
-0.0126622252088158, -0.0421199498823812, 0.0413010985037516,
0.117248951520421, 0.0585658398394289, -0.045854748998357, -0.0776621731946507,
-0.0109836217915529, 0.0301421839292724, -0.0477608577296227,
-0.0518357302381656, 0.0872367662803821, 0.168199146272765, 0.0835226405276321,
-0.0431290697072093, 0.039000331547431, 0.0904456574959092, -0.0891636664849909,
-0.148245088843424, -0.0514561657081618, 0.00627334028739845,
0.00932653803506511, -0.036714228160621, 0.00401303926886598,
-0.0326019391733094, -0.0272955247353401, 0.0407768868811118,
0.0756503032722406, 0.0232879385327998, -0.0356623156999039,
-0.0182042162604343, 0.012405297305115, -0.0311314558844096,
-0.0757327062857556, -0.0438842167278324, 0.0534484315397847,
-0.00263093389834701, -0.0407968808622812, 0.0513899218257041,
0.161084849737891, 0.10414140315619, -0.105434010860225, 0.0317302108906947,
0.105366400083279, -0.256112774850456, -0.151872260620654, -0.0476317463908522
)), row.names = c(NA, -110L), class = c("tbl_df", "tbl", "data.frame"
))
If the problem is to find where the fit column starts rising then fit a curve made up of a horizontal line segment followed by a sloped line segment (red in the graph) and report the changepoint (Date0 and dashed line in graph).
# calculate starting values, st
fm0 <- lm(fit ~ Date, data, subset = seq(to = nrow(data), length = 20))
st <- c(mean(data$fit[1:20]), coef(fm0))
names(st) <- c("a0", "a", "b")
fm <- nls(fit ~ pmax(a0, a + b * as.numeric(Date)), data, start = st)
# solve a0 = a + b * Date0 for Date0 using calculated a0, a and b
Date0 <- with(as.list(coef(fm)), .POSIXct((a0 - a)/b))
plot(fit ~ Date, data, ylab = "")
lines(fitted(fm) ~ Date, data, col = "red")
abline(v = Date0, lty = 2)
Date0
## [1] "2017-03-21 07:53:56 EDT"
Related
I was trying to use the solution offered here to find all the location from df which are within the 70 km distance from my point of interest userLocation=c(6.9,55.2), but it does not work properly !
df = structure(list(lng = c(6.2694184, 6.25737207, 6.23839104, 6.25844252,
6.22595901, 6.21351832, 6.2010845, 6.1886414, 6.1762058, 6.1637609,
6.15132287, 6.13887619, 6.12643637, 6.14361895, 6.16332364, 6.18302157,
6.2027276, 6.22242688, 6.24213488, 6.26842752, 6.26745135, 6.24518597,
6.26645948, 6.24420242, 6.22357831, 6.26548171, 6.24321746, 6.2226023,
6.20041884, 6.18070459, 6.16099845, 6.16716672, 6.17960629, 6.18686265,
6.2078525, 6.19203657, 6.20447434, 6.21691835, 6.2293537, 6.24179593,
6.26009321, 6.26448764, 6.2422317, 6.21927538, 6.20186455, 6.26350828,
6.24124514, 6.22028969, 6.26251321, 6.2402584, 6.23404584, 6.26153227,
6.22171658, 5.94065657, 6.10363006, 6.11606487, 6.12850589, 6.14093826,
6.15337749, 6.16582359, 6.17826103, 6.19070472, 6.20313974, 6.20009703,
5.96044213, 5.96988333, 5.98023582, 5.98966667, 5.99910246, 6.00003829,
6.00947365, 6.01889843, 6.02832882, 6.01983402, 6.02925771, 6.038687,
6.0481219, 6.05754688, 6.03963788, 6.04906608, 6.05848435, 6.06792377,
6.07735326, 6.08677283, 6.05941948, 6.06885218, 6.07829049, 6.08771889,
6.09713671, 6.10657633, 6.11600538, 6.07922538, 6.08864707, 6.10756108,
6.12000483, 6.13243993, 6.12019786, 6.14488189, 6.15733073, 6.16977091,
6.16621949, 6.13805015, 6.13652024, 5.941545, 6.20491484, 6.18423897,
6.17806466, 6.16355552, 6.15738696, 6.14558294, 6.14286638, 6.13670293,
6.12217027, 6.11601258, 6.10148275, 6.09533146, 6.08080511, 6.07464337,
6.06011984, 6.03729438, 6.05394895, 6.02546329, 6.0136389, 6.03674112,
6.05743408, 6.07812006, 6.09879971, 6.11948795, 6.11063647, 6.08914275,
6.08440881, 6.0018212, 6.02491713, 5.98999461, 6.01308427, 5.97815849,
6.00125809, 5.96632973, 5.98943792, 5.9995124, 6.02119838, 6.04364466,
6.0223476, 6.04560587, 6.03821257, 6.06131821, 6.06046748, 5.97888909,
5.95766873, 6.24771247, 6.04931495, 6.25538943, 6.23227728, 6.25434093,
6.25329159, 6.25225759, 6.25120656, 6.25015469, 6.24911757, 6.06338238,
6.08539205, 6.10756976, 6.12975108, 6.15193667, 6.17411029, 6.19630377,
6.21848591, 6.22602495, 6.23123663, 6.20931486, 6.23019515, 6.20826628,
6.22915282, 6.20721685, 6.22810966, 6.21962063, 6.20209266, 6.20618216,
6.19702482, 6.1799057, 6.15772301, 6.13554395, 6.11336914, 6.09118237,
6.09738412, 6.11958004, 6.12698723, 6.14767387, 6.16835417, 6.18613747,
6.185096, 6.165456, 6.14476821, 6.15765091, 6.23561071, 6.08001353,
6.22353732, 6.2376767, 6.21143885, 6.19936347, 6.18727866, 6.17520066,
6.16311385, 6.15103386, 6.13894506, 6.12686243, 6.11478725, 6.10270261,
6.09818625, 6.12128852, 6.2468456, 6.22571713, 6.24558662, 6.22445138,
6.24434288, 6.22320086, 6.24308194, 6.22194875, 6.24182062, 6.22068065,
6.24057332, 6.21942655, 6.2113264, 6.22341814, 6.19699748, 6.18490568,
6.1988361, 6.17283631, 6.16074252, 6.14867115, 6.13657473, 6.13954049,
6.16263694, 6.18482009, 6.20327221, 6.20009595, 6.19278885, 6.17005571
), lat = c(54.67598304, 54.83924292, 54.83162024, 54.82483795,
54.82033259, 54.80904336, 54.79775292, 54.78646988, 54.77517665,
54.76389082, 54.75260377, 54.74131515, 54.73002531, 54.72096456,
54.71392047, 54.70687309, 54.69983176, 54.69278713, 54.68573957,
54.68934722, 54.7027117, 54.69910571, 54.71606682, 54.71246092,
54.70614626, 54.72943123, 54.72582507, 54.71951053, 54.71576339,
54.72280423, 54.72985112, 54.74274399, 54.75402944, 54.73569581,
54.72983408, 54.7653223, 54.77660496, 54.78789538, 54.79918423,
54.81047187, 54.80230996, 54.74279524, 54.73918917, 54.74155047,
54.75043676, 54.75615956, 54.75255324, 54.75849353, 54.76951451,
54.76590829, 54.77879358, 54.78287875, 54.84106585, 54.79004116,
54.73264696, 54.7439301, 54.755221, 54.76651031, 54.77779842,
54.78908531, 54.80037062, 54.81166369, 54.82295519, 54.83631649,
54.78306731, 54.79535153, 54.77609951, 54.7883729, 54.80065457,
54.76912877, 54.78140068, 54.7936805, 54.80595963, 54.7621547,
54.77443373, 54.78671208, 54.79898973, 54.81126633, 54.75518666,
54.76746422, 54.77974071, 54.7920169, 54.80429202, 54.81656608,
54.74821493, 54.76049101, 54.7727664, 54.78504073, 54.79732298,
54.80959594, 54.82187682, 54.74124062, 54.75351485, 54.76118719,
54.77247897, 54.78376916, 54.79513973, 54.79505815, 54.80634591,
54.8176321, 54.8309898, 54.82587825, 54.81251828, 54.80340625,
54.85043439, 54.85669012, 54.84379843, 54.8629602, 54.85006754,
54.83850747, 54.86921769, 54.85633303, 54.87548056, 54.86259492,
54.88174916, 54.86885358, 54.88800555, 54.8751176, 54.89427628,
54.89688611, 54.88137801, 54.88534048, 54.87379377, 54.87235508,
54.86608859, 54.85982748, 54.85356275, 54.84730376, 54.83491117,
54.82992904, 54.84309445, 54.86224596, 54.86080945, 54.85069668,
54.84926234, 54.8391549, 54.83771415, 54.82760306, 54.82617384,
54.81401612, 54.81866848, 54.82193287, 54.83203, 54.85454492,
54.8418856, 54.84463218, 54.83126946, 54.807705, 54.81314447,
54.95082492, 54.90870481, 54.85261135, 54.8544958, 54.86597391,
54.87933643, 54.89269028, 54.90605271, 54.91941509, 54.93277779,
54.91930405, 54.92337651, 54.92712528, 54.93087901, 54.93462874,
54.93838307, 54.94213376, 54.94588009, 54.93325056, 54.86785843,
54.86365241, 54.88122101, 54.87701472, 54.89458354, 54.890377,
54.90794604, 54.9204011, 54.92916856, 54.90373959, 54.91606989,
54.92541899, 54.92166542, 54.91791682, 54.91416421, 54.91041621,
54.89753741, 54.90128459, 54.88863077, 54.8823668, 54.87609923,
54.88468492, 54.89804726, 54.89094652, 54.89720452, 54.90830415,
55.08370977, 54.93641839, 55.07226944, 55.06170442, 55.06083624,
55.04939327, 55.03795778, 55.02652115, 55.01508302, 55.00364374,
54.99220296, 54.98077001, 54.96932695, 54.95789135, 54.94469353,
54.94333719, 54.96418243, 54.9585686, 54.97754901, 54.9719349,
54.99090692, 54.98529252, 55.00427341, 54.99865908, 55.01763087,
55.01201625, 55.03099761, 55.02538271, 55.03790915, 55.04934232,
55.02208092, 55.01064489, 54.99998971, 54.99920808, 54.98776941,
54.97632995, 54.9648976, 54.95153594, 54.95007267, 54.95382515,
54.96186591, 54.98662348, 54.97394968, 54.97118391)), class = "data.frame", row.names = c(NA,
-238L))
What I have done is as follow :
Add the point of interest to the beginning of df
df = rbind(userLocation,df)
Set the radius to 0.64 since according to here, every 0.1 is equivalent to 11.1 km !
radius <- 0.64
#Identifying neighbors
res <- nn2(df, k=nrow(df), searchtype="radius", radius = radius)
Since my point of interest is the first row in df I would expect all the non zero index in the first row are the points within my 70 km threshold
Ind <- res$nn.idx[1,][res$nn.idx[1,]>0]
My Ind object has just one value!
Ind
[1] 1
but if I plot the data, all of the points are within 70 km distance :
I would appreciate it if someone could help me here.
I have downloaded some raw data on inflation which said in the description that it has not been seasonally adjusted. Therefore I assumed that there is some seasonality present. I did not check if this is indeed true. What is the common way of checking a time-series for the presence of seasonality?
Nevertheless, I still tried to apply the procedure, but the before and after result are exactly the same; nothing has changed.
The code:
if (!require("tis")) {install.packages("tis"); library('tis')} # Load time series library
if (!require("seasonal")) {install.packages("seasonal"); library('seasonal')}
inflation.start <- c(1960,1)
inflation.end <- c(2018,1)
inflation.raw <- "rawData/germany_inflation.csv"
inflation.table <- read.table(inflation.raw, skip = 1, header = F, sep = ',', stringsAsFactors = F)
inflation.ger <- ts(inflation.table[,2], start = inflation.start, frequency = 4)
# ts.plot(inflation.ger)
inflation.seasadj <- final(seas(as.ts(naWindow(inflation.ger),freq=4))) # seasonal adjustment
inflation.seasadj.ger <- ts(inflation.seasadj, start = inflation.start, frequency = 4)
So apparently, inflation.ger and inflation.seasadj.ger are the exact same time-series. What could have gone wrong?
The data:
dput(inflation.ger)
structure(c(2.22222222222224, 1.244019138756, 0.75973409306742,
1.80608365019013, 1.98487712665404, 2.64650283553874, 2.73327049952876,
3.36134453781511, 3.15106580166824, 2.39410681399631, 2.47706422018348,
3.25203252032522, 2.87511230907457, 2.42805755395685, 3.31244404655327,
2.27471566054242, 2.09606986899562, 2.72168568920105, 2.2530329289428,
2.05303678357573, 3.07955517536358, 3.84615384615386, 3.98305084745763,
4.10729253981556, 4.06639004149377, 3.04526748971191, 2.93398533007336,
2.49597423510466, 1.8341307814992, 1.75718849840257, 1.10847189231985,
1.49253731343286, 1.25293657008615, 1.33437990580847, 1.80109631949884,
1.62538699690402, 2.01082753286931, 2.09140201394268, 1.92307692307691,
2.89413556740292, 3.33586050037907, 3.5660091047041, 4.00000000000001,
4.44115470022204, 5.06236243580337, 5.78754578754579, 5.66037735849054,
5.31537916371368, 4.88826815642457, 5.47091412742381, 6.25000000000001,
6.46029609690444, 7.39014647137152, 6.9599474720946, 7.30446024563674,
7.45891276864728, 7.06757594544325, 6.99815837937383, 6.44578313253009,
5.88235294117648, 6.25361899247252, 6.08146873207113, 5.43293718166383,
5.22222222222223, 4.30517711171662, 3.78583017847487, 3.70370370370372,
3.80147835269271, 3.81400208986417, 3.85617509119333, 3.46790890269152,
3.00101729399797, 2.86864620030195, 2.50878073256398, 2.50125062531265,
2.91358024691356, 3.27788649706457, 4.650024473813, 5.31966813079548,
5.51823416506719, 5.92136428233066, 5.19176800748362, 5.14365152919371,
5.72987721691677, 5.81395348837209, 6.66963094708762, 7.13970912296167,
5.89247311827958, 5.36770921386304, 5.08545227177989, 4.648292883587,
4.06173842404549, 3.28920978740473, 3.13367711225704, 2.71226415094341,
2.84933645589384, 2.83495145631071, 1.88461538461538, 2.06659012629162,
2.31499051233397, 2.30362537764349, 2.076255190638, 1.57480314960629,
0.741839762611279, 0, -0.332840236686399, -0.922849760059066,
-0.515463917525782, 0.0369139904023732, 0.519480519480511, 0.968703427719836,
0.962250185048109, 1.14391143911439, 1.29198966408267, 1.69741697416972,
2.4193548387097, 2.88215979569498, 2.80612244897959, 3.01161103047896,
2.72011453113812, 2.3049645390071, 2.72952853598016, 3.02923564635434,
2.84251480582282, 3.29915972330998, 4.51351791825871, 5.50437317784258,
5.9108527131783, 6.09404990403073, 4.95750708215298, 3.33642261353105,
4.66605672461115, 4.38715513342377, 4.5434098065677, 4.30493273542601,
2.97202797202796, 2.77296360485268, 2.58175559380379, 2.45055889939811,
2.07979626485568, 1.7284991568297, 1.5520134228188, 1.46873688627781,
1.45530145530146, 1.45047658516367, 1.36307311028499, 1.53019023986766,
1.76229508196721, 1.55228758169937, 2.32273838630808, 2.11812627291242,
1.24848973016513, 1.36765888978277, 0.597371565113501, 0.438771439968094,
0.198886237072386, 0.515873015873038, 0.673000791765629, 0.953137410643369,
1.54823342596267, 1.10540860639556, 1.37632717263074, 1.73092053501181,
1.72009382329947, 2.53807106598985, 2.01706749418153, 1.6627996906419,
1.99846272098384, 1.29474485910129, 1.21673003802282, 1.17915557246102,
1.13036925395631, 0.789473684210527, 1.05184072126221, 1.16541353383457,
0.968703427719814, 1.86497575531517, 1.82156133828996, 2.0066889632107,
1.62361623616237, 1.24496521420726, 1.64293537787514, 1.67577413479055,
1.70660856935367, 1.84448462929475, 1.4727011494253, 1.28986026513793,
1.7850767583006, 2.05965909090908, 2.26548672566371, 3.0774672798019,
2.94633461943178, 2.88796102992346, 3.08065074420214, 1.61290322580647,
0.81771720613288, 0.270544470747383, -0.235057085292139, 0.405268490374862,
0.811084825954715, 1.11298482293424, 1.11073712554695, 1.37907837201482,
1.87730472678512, 2.00133422281521, 2.19707057256989, 2.22295952222959,
2.13886146758801, 1.86396337475473, 2.01954397394136, 2.01233365790325,
1.54639175257732, 1.50882825040129, 1.62835249042147, 1.33630289532296,
1.20558375634519, 1.07526881720429, 0.848256361922726, 0.50235478806906,
0.0313479623824208, 0.469336670838559, 0.124610591900319, 0.312402374258052,
0.250705108116592, 0.0934288383680993, 0.466708151835709, 1.12114606041732,
1.87558612066271, 1.68014934660859, 1.73428305977082, 1.66307360640589,
1.50352868978214), .Tsp = c(1960, 2017.75, 4), class = "ts")
We have time series data with some seasonality from the past 4 years. We want to predict the general rise in trend next year. For this, we decomposed the time series and observed the trend line:
However, this trend line is placed in the middle of the values rather than the values themselves. We are not satisfied with simply extrapolating this trend line since it falls very short of expected traffic.
Since we are interested in only the general rise in trend and not the seasonality, we remove the seasonality from the components:
mydata <- read.csv("values.csv")
mydataseries <- ts(mydata, start=c(2012,1,1),frequency = 365.25
mydataseriescomponents <- decompose(mydataseries)
mydataseriesminusseasons <- mydataseries - mydataseriescomponents$seasonal
We are now trying to fit a loess() curve in R around this time series data minus the seasonal component, that is mydataseriesminusseasons:
855154881.9
1027395443
1132284155
944870172.3
898459083.9
845115286.7
204393180.2
-75788428.32
-184120868.7
-164634776
-190543808.7
-43973009.39
-452418843.2
-1065106918
-1194545584
-1250333168
-379435027.4
-151057609.1
134304962.6
37020062.65
-307740042.8
-309480234.5
-388529539.7
-379333445.3
-193124460
-663765015.3
-100597898.3
-327949890.1
-429500583.5
-1321506072
-1444202356
-369913100.7
-715237274.1
-83507361.25
-328296509.9
-409957935.1
-1351211680
-533631519.6
-882845870.8
-711202595.9
-147504975.3
-361087020.5
-392960561
-456709751.2
-1065209981
-2062128692
-1112973164
-914751976.5
-1377070656
-1903432323
-2110392180
-1984477186
-1434481986
-988902503.8
-944477878.3
-1388889111
-1559730054
-1668752977
-1590459585
-1528272998
-850923022
-542720361.8
-791860111.9
-1057854181
-852120155.6
-810625768.1
-56185632.59
576968488.7
691371629.9
160486150.2
-308533580.1
-261619133.2
-404620378.3
740114544.5
1006206606
622867052.1
-71219884.82
-39173777.42
-533631907.8
-709943766.3
-565909169.9
-968499911.2
-1056539190
-1266350530
-1445031918
-1579156208
-1550356091
-1316401212
-899907725.4
-681646243
-996858524.4
-1261573594
-1620842348
-1529532466
-1107218074
287538766
303061371.4
-382291371.7
-639757694.3
-737369705.5
-732118809.2
-393825621.1
-810602326.3
-1044759123
-1410535140
-1636325717
-1646415420
-1514426422
-1241825553
31448256.14
64762654.11
-513468522.6
-676115681.9
-191498426
-86743267.18
80585792.24
538395944.8
262807408.8
-156283609.7
-72157485.31
-56386961.34
-14257457.85
233362524.8
495465812.9
354441383.3
74975465.29
28985451.46
187381822.2
517857439.4
940857149.4
1397097814
1965769992
1735207534
1568036010
1496104390
1647402384
928936108
1372629124
1417356139
1168474219
1080545493
996524274.5
988026158.1
1060004940
1315522041
1444552768
1438503151
1174067300
1063743311
1086528293
1310565938
1542624917
1504331433
1280216655
1726223821
1670308928
1120574360
1084897870
1285908119
1232985139
1101293494
1081075441
881717495.1
975005126.7
1058436909
1290433280
1288259727
1038856967
1006240656
891910425.2
920601850.4
981585788.8
1536507833
1546018200
1149902811
1230460988
1191817042
1161287112
1273239958
1590385882
1538451838
1242359113
913747727.7
888013837.2
934884821.1
1198395868
1277044767
1235456131
963043028
925500611.4
921906159.7
957528730.9
934517634.5
1183155135
1155142736
893512199.3
829955361
839182104.7
904102148.6
972879836.6
1227504663
1192420215
904137203.4
944756516
1375037708
1328414157
1779424344
1921248556
1899629843
1773839270
1565431502
1592386775
1825650269
1404794363
1393784791
1416771770
1288386476
1289239680
815093905.4
681990799.1
783953618.4
1103067833
1182429773
933897907.8
874114515.6
787403408.5
860338795
1052657878
1297180989
1245476158
1014316130
1004606484
735958898.4
575505161.4
588750452.3
803692654.7
833093724.3
531996754.3
199110366.8
-18901281.1
-6413198.146
-33290860.64
252073940.6
203247618.9
-164091120.7
-501720846.4
-865543851.3
-856086326
-612355346.3
400846624.1
511862726
288735892.8
-127533264.3
-940372573.2
-155917893.1
-699989731.6
-320401187.5
-277824927.9
-705440125.7
-1042182377
-1206106831
-1339738778
-165282703.3
222304361.4
136773204.1
-224139914.3
-271723348.1
-288223577.8
-285482682.4
-1057479814
-1059799528
-960773269.6
-1378106942
-1682462197
-1880270405
-1877392694
-1621410466
-1218560480
387109224.3
85424610.03
-558048286.6
-432347510.8
-107370046
224698124.4
732906395.5
-523071744.4
-1239504492
-1798916179
-2065361083
-1987860160
-1725292219
-1109123667
-971275176.7
-1420498329
-1889158526
-1136864728
-1002777887
647298007.9
1277421457
248465769.8
-377527211.2
-724681515.2
-967478152.4
-947666627.1
-734506065.3
-408053802.2
-241421840.8
-363715586.3
-829237162.5
-929529173.2
-260432788.4
-67756618.13
242997207.2
567370574.9
128921994
-159670249.1
-226982681.8
-240623081
-26502753.8
311495394.1
477285225.3
198408156.8
-195753985.4
-271887709.5
-196538766.7
-72464893.44
180860168.9
360960891.9
147945699.2
-29352193.17
325127464.9
960490365.6
1453481987
1424677454
1207446280
537281902.1
-167977127.5
-505612366.5
-1460096451
-1554214240
-1472813276
-1158137635
-1079634747
-1400116741
-1623247577
-1720625509
-1291068926
-541823228.2
423680238
828606261.2
651175802
588260033.4
511022741
644493024.6
696908085.1
949150687.9
1037411745
754948966.3
790033725.6
1053226989
1170333305
1056602964
1081413405
1135864583
971795777.8
855154881.9
1027395443
1132284155
944870172.3
896319497.9
845115286.7
204393180.2
-75788428.32
-184120868.7
-164634776
-190543808.7
-43973009.39
-452418843.2
-1065106918
-1194545584
-1250333168
-379435027.4
-151057609.1
139705076.6
37020062.65
-307740042.8
-309480234.5
-388529539.7
-379333445.3
-193124460
1308430606
-100597898.3
-327949890.1
-429500583.5
-1321506072
-1444202356
-369913100.7
1419976593
180125360.7
-328296509.9
-409957935.1
-1351211680
-533631519.6
1246061777
1639641696
272442819.7
-361087020.5
-392960561
-456709751.2
1014735303
478499133.8
-748455756.5
-914751976.5
-1377070656
-1903432323
-2110392180
-1984477186
-1434481986
-988902503.8
-944477878.3
-1388889111
-1559730054
-1668752977
-1590459585
-1528272998
-850923022
-542720361.8
-791860111.9
-1057854181
-852120155.6
-810625768.1
-56185632.59
576968488.7
681069818.9
160486150.2
-308533580.1
-261619133.2
-404620378.3
740114544.5
1006206606
622867052.1
-71219884.82
-39173777.42
-533631907.8
-709943766.3
-565909169.9
1641890896
1588111061
880069519.6
1090752417
1278890698
1335767912
1519822543
1812984581
1334545973
426228457.6
456951496.9
1228693380
1312355639
1637418032
691968559
303061371.4
-382291371.7
-639757694.3
-737369705.5
-732118809.2
-408383535.1
1630144553
1548715666
747397223.2
884268460.1
1205537489
1226277745
1637844137
477586396.1
64762654.11
-513468522.6
-676115681.9
-191498426
-86743267.18
80585792.24
538395944.8
262807408.8
-156283609.7
-72157485.31
-56386961.34
-14257457.85
233362524.8
495465812.9
354441383.3
74975465.29
28985451.46
187381822.2
517857439.4
940857149.4
1397097814
1479492584
1206235886
1230335054
1209244303
1772074811
1693077942
1461745247
1417356139
1168474219
1080545493
996524274.5
988026158.1
1059643199
1314652245
1440434637
1438503151
1174067300
1063743311
1086528293
1310565938
1542624917
1504331433
1280216655
1726223821
1670308928
1863627651
1882123871
2055914335
1992301669
1579375771
1533925079
1762629244
1838997943
1897919645
2107505072
1997289608
1464337865
1415561153
1676756233
1700040307
1783398157
1624554189
1546018200
1149902811
1230460988
1191817042
1161373475
1273239958
1590385882
1538451838
1242359113
913747727.7
888013837.2
934884821.1
1198395868
1277044767
1235456131
963043028
925500611.4
921906159.7
957528730.9
934517634.5
1183155135
1151401692
893512199.3
829955361
839182104.7
904102148.6
972879836.6
1227504663
1192420215
904137203.4
944756516
1375037708
1328414157
1193178073
1253502729
1251928059
1185271006
1244858293
1231640212
1171269572
1343448619
1393784791
1416771770
1288386476
1289239680
815093905.4
681990799.1
783953618.4
1103067833
1182429773
933897907.8
874114515.6
787403408.5
860338795
1052657878
1297180989
1245476158
1014316130
1004606484
735958898.4
575505161.4
588750452.3
803692654.7
833093724.3
531996754.3
199110366.8
-18901281.1
-6413198.146
-33290860.64
252073940.6
203247618.9
-164091120.7
-501720846.4
-865543851.3
-856086326
-612355346.3
400846624.1
511862726
288735892.8
-127533264.3
-940372573.2
-155917893.1
-699989731.6
-320401187.5
-277824927.9
-705440125.7
-1042182377
-1206106831
-1339738778
-165282703.3
222304361.4
136773204.1
-224139914.3
-271723348.1
-288223577.8
-285482682.4
410687500.7
1631841798
1553281634
713309450.5
649367763.9
960215164.3
1049107976
1148989485
1678375251
1164709303
183895982
465208739.4
766073381.2
586658157
853178118.4
1317822533
1674419406
693138111.6
610057420.2
928861116.6
1021382975
1317392280
1950890283
1753467250
679132043.8
530667904.1
1923321662
2445945599
2224779498
2205640478
2617939193
1771677175
1773936492
2444894838
2401384689
2491268212
2581667225
2449579792
1593738389
1956098689
2378730864
235226517.6
-67756618.13
242997207.2
567370574.9
128921994
-159670249.1
-226982681.8
-240623081
-26502753.8
311495394.1
477285225.3
198408156.8
-195753985.4
-271887709.5
-196538766.7
-72464893.44
180860168.9
360960891.9
147945699.2
-29352193.17
325127464.9
960490365.6
1453481987
1424677454
1207446280
537281902.1
-167977127.5
-505612366.5
1404374892
1613823539
1849271413
1936762445
1513383047
1256267888
1553901607
1655449233
1979063830
2224098910
2147758870
1612601292
1277373178
1489383946
1406268621
1510782787
1555473157
1721503681
1473734831
1214127421
1427159815
1454163770
1559572239
1693448937
1686056516
1534649610
1402815411
1597472633
1479661974
1572474919
1758527502
1912156129
2007806540
2577055685
3079863351
3189154952
3084552356
3020184398
3019956688
2361199448
1906290720
1878596097
1894542086
54727454.63
-151057609.1
134304962.6
37020062.65
-307740042.8
-309480234.5
-388529539.7
-379333445.3
-193124460
-663765015.3
-100597898.3
-327949890.1
-429500583.5
1538626202
1943756793
142960155.3
-715237274.1
-83507361.25
-328296509.9
-409957935.1
1469349471
-6715352.565
-882845870.8
-711202595.9
-147504975.3
-361087020.5
-392960561
-456709751.2
-1065125015
1006714631
2153142496
1804378762
1673270778
2693842453
3029003743
3091803034
2506604564
2457560591
2027821010
1756470132
2064844611
2119623546
2162263280
2002164168
1694641809
616667238.2
178971439.1
187991217
342780020.4
452558045.9
1221341191
1928282804
1898438088
2350801273
3271266186
3102579205
3404316683
1296736764
1006206606
622867052.1
-71219884.82
-39173777.42
777976197.2
831608580.7
1173972977
574093131.8
145884449.2
125630390.6
140647407.2
-1948845.745
-20569346.72
170376510.8
416953573.6
394780276
210086267.6
240846310.9
-4692367.437
13267344.2
462459409.2
1638331860
1368251762
844946221.3
908678722.7
858991700.5
821130809.8
1087001122
501346560.7
49030467.37
-115293555.8
-42916860.95
-94506694.82
65744337.46
214422930.7
1246493179
948746467.1
450984956.4
906168925.1
55072855.99
-86742286.18
80586773.24
538396925.8
262808389.8
-156282628.7
-72156504.31
-56385980.34
-14256476.85
233363505.8
495466793.9
354442365.3
74976446.29
28986433.46
187382803.2
517858420.4
940858130.4
1397098796
1479493565
1206236868
1230336035
1209245284
1021481495
1358007248
1908140672
1690817796
1450430426
1573572004
1548289864
1518408336
1551340078
1800682764
1691240562
1675755235
1422581675
1551622923
1630779709
1361558060
1542625898
1504332415
1280217636
1726224802
1670309909
1120575341
1084898851
1285909100
1226741796
1101294475
1081076422
881718476.1
975006107.7
1058437890
1290434261
1288260708
1038857948
1006241637
891911406.2
920602831.4
981586769.8
1536508814
1546019181
1149903793
1230461969
1191818023
1161288092
1273240939
1590386863
1538452819
1242360094
913748708.7
888014818.2
934885802.1
1198396849
1277045748
1235457112
963044009
925501591.4
921907140.7
957529712.9
934518615.5
1183156116
1151402673
893513180.3
829956342
839183085.7
904103129.6
972880817.6
1227505644
1192421196
904138184.4
944757497
1375038689
1328415139
1193189782
1253504218
1251929547
1185272494
1244859781
1231641700
1171271060
1343450107
1393786279
1416773258
1288387964
1289241168
1276942382
1223342782
1265179173
1582950274
1549362846
1289741511
1522844687
1575762386
1623981295
1140018366
1297182477
1245477646
1014317618
1004607972
3339996446
2617836416
2598501271
2725463672
2379954942
2430902194
3387783330
4057325882
3941464221
4026010913
3593315188
3218647755
2899429341
3156298547
3453939489
3535652872
3957490786
2127115167
1652343230
1644069350
1408460965
3374914231
1354451507
3451778958
4125068975
3062203905
2860803149
3523217492
3446260128
4086491216
1353224655
1786887688
1329952369
1262946794
1434089723
1426765711
1394483892
3184736742
3585370070
2492338381
2870647454
3422458727
3496074936
3440427221
3488297885
3411694273
1862226333
2347786389
2984492965
2832984919
2706245540
2975365013
2750770215
2329108891
2668972592
3485974677
3498308730
3651172133
4020199587
3953317903
2733969884
2903235323
3953139670
4668898648
4652525612
4073702669
3715845753
3017343417
4181519935
4751974434
4619926728
4594101972
4732221475
4318140179
3780497707
4451918052
5012718317
4833179055
5653878208
5807608123
5396558840
4198373572
4884875793
5677377778
5912605893
5944449107
5747882280
5227572313
4478777171
4801671568
5862958412
6148995450
6002089979
5948770595
5519491673
4863435104
5079255850
5868770156
5542293262
4248733096
2766041977
2732273377
2763007960
3709017196
5826881003
5418281457
4464795375
4353979464
4041806357
3642750843
3991233296
4574288639
4505497973
4507525662
3759850718
2770466566
1884370202
2017997394
2536894291
2394368239
2561217637
2329522814
2238153751
1823017685
1985828850
2534460960
2292248472
2008277139
1792241233
1777656572
1763197595
1863541282
2165268053
2093475484
2045555081
1853852293
1858329332
1755509088
1713779365
1790957115
1574452597
1579659809
1670558649
1766876466
1623856630
2694719637
3768836252
3933201414
3980190714
4955559887
4938681336
4365493305
4568797166
5264823884
5274383182
5433866314
5417386015
5045165577
4013353224
4858457348
5313474135
5517720717
4442688396
4160415512
4888461743
3958104821
4567324340
5323056776
5488909281
4557331369
5223549360
4326783309
3764876526
4571173289
5423886480
5493146693
5626783987
4769215133
3238864341
3322242772
3840761593
4440231932
3952228746
3826102990
3642814617
3683433336
3290239564
3679644227
4401886050
4271007289
4331924299
4217934600
4322830334
3958867188
4735975510
5434096780
5704487461
5359613448
5223744170
3712806728
2381705672
2316848053
2396344551
1960261326
2098710139
1955760687
2933140424
2967171559
3746036475
5150516737
5103739176
4799974487
4940976107
4470787625
2874748403
3370617116
4116000776
3907376783
3879136871
3858305272
3498190930
2968939532
3482738547
4592541370
4548237451
4039300654
3954893868
3196099484
2977703338
3645589701
4866174649
5072864386
5232041565
5275606802
4693688082
3245094491
3790221336
4772983985
4581973930
4331144672
4167962637
3622317542
3760217425
4483667233
5577532419
5303025177
5677698961
5720893489
5385575901
4466914817
5021515141
5865735715
5703914224
5678717816
5600943246
5110447244
4590128626
4877035122
5442054864
5540200590
5229513831
4574668437
3734155348
2825747731
2665659550
3218611753
3177069031
3225833078
2857378098
2605242926
2290489544
2556941031
3052363819
3023655743
3139325913
3183875762
3084641745
2584707861
2573111682
2596449197
3119292036
3103807890
3003440326
3050225234
2639037382
2719271674
3174072606
2288642761
2407051251
2769659370
2792708736
2422258872
2550055714
3094896689
3167462954
3145311598
2982505987
2846929030
2410677820
2528516371
3317404595
3405548963
3264859256
3217887704
3076794348
2682319143
2752574869
3546644862
3387139453
3465822066
3527697761
3304648740
2671176465
2775904656
3369489444
3186975051
3265608930
3125970955
2336383605
2101672532
2227649209
3046717349
3161392531
3174174168
3069526988
3140929673
2396972936
2491933483
3274873818
3471396399
3449881421
3260944652
3059846681
2300326480
2410144424
3280639160
3165080129
1880734112
2026731745
1851573264
1593311358
1623039610
1888090770
1983714225
1990135314
1884610261
1966858510
1881518298
1817889199
2209334095
2213195958
3180360773
3506589476
3266458246
2314386847
2203488644
2979775043
2887785856
3029641480
2854946175
2971200632
2338656676
2508827357
3221768984
3271647981
1494100499
2557318665
2566743784
2024262236
2326858931
2898711691
2629767554
2418419575
2529673112
2517326919
2394071701
2882872031
3958018274
4398615549
4850186215
4769908627
3875047506
3686152334
3944845885
4407300286
5483421647
5150638137
5610203814
4608824221
3183456836
4173041229
5250468129
5281994707
5707036828
5352424541
5755606494
4561838923
5206514812
6017273753
5965257129
6031374315
6079040231
4379684817
2770833772
3855818791
4754914912
4592227424
4426284753
4410453960
4025381538
3174239722
3637497533
4441638488
4174099821
3906145103
3894032411
3032325960
2296152859
3634154800
5013642209
4862468183
4819578542
4516969809
3607317085
2438327317
3732908830
5107431888
4696140901
1856348267
1237864483
406427050.9
166776835.3
1498808575
1828048545
1624152578
1349901570
1420774156
998226407.7
1012022910
1532830520
1859891754
1422784584
1300680502
1969725485
1936125754
1730921235
2284540775
2480696530
2271292433
2177291835
2174315253
1943550394
1789789173
2208387982
2445973421
2180115879
2053590351
2057128187
1868753587
1794479025
2093710971
2310482091
1882157273
1506021669
1535469156
2038211764
2132884419
2539631965
2937476532
2237006540
1897259543
801631140.7
589473184.2
116334380.3
674275361.3
1282712559
1119202007
1004676722
781599593.7
1184057471
1405174402
2414122535
3215047803
3117104932
2843730513
2338714670
1830982498
1282357057
1398445510
1709185553
1343346425
1299455992
1382754660
1451688601
1355952002
1395566041
1646971917
1518132724
1378268909
1383505613
1477733701
1396182586
1340692285
1513095775
952400088.2
654084246.7
626929432.3
700786606
523798987.3
1212948425
1567710490
1072360130
1057312922
876650104.6
1532682661
1439342205
1801524855
2099113224
1992935512
2063753470
1849349346
1589462235
1341435771
1147947080
2071354378
2006092710
2105408195
1234206054
656144711.4
1208902786
1232216105
2213555415
2017146411
1991371378
994748835.5
1523018258
551904050.2
4489317425
7011882149
7509029269
7405531928
6627086822
5118361296
2929515389
4386298474
6193577031
6201888044
5600929516
5249637157
4071570693
3812427933
4814958529
6581889904
6706593436
6538328968
5770674289
5146640592
3869631764
5168489883
7218970135
7319257013
6818980598
6581026441
4463128268
2549197744
2751600290
3586655201
3073231312
2610229695
2664878334
2445857027
3333286739
5342397336
8218514365
7400486992
7351969323
6170498161
6166815740
4938259613
5123232548
6952344511
7484834076
7975757356
6042108897
3954139391
2561409558
3636287452
6659637045
6610145475
6303751643
5790815423
5286756045
532871596.2
2403662106
2933984545
2177296758
1954486433
1713519574
1459802901
4020142932
5383799348
6858179971
7112822894
6327294007
5759923625
5042533123
3796835736
5750880779
7453820307
7335397513
6962686301
7790660114
6724956363
5003169135
6701050938
7992041249
7756326177
7913301179
7741411043
6785177818
6284270117
6963921883
7785974652
7574222100
7588499293
5788919572
4105788142
2859825648
2924210471
4069907537
3895077405
3979654780
2075329893
1979403277
1632081713
1877890303
2412917344
3763581680
3726439732
3673898152
3345262416
2191053212
2548906553
3841133834
3846853011
3492242728
3479185981
3290714847
2514060721
2645599322
2791855040
3791586130
However, we are not familiar with the form of the loess() function and it seems to expect a formula which we do not know. We tried the following:
y <- c(mydata)
x <- 1:1614
lo <- loess(y~x)
We get the error: Error in model.frame.default(formula = y ~ x) :
invalid type (list) for variable 'y'
How do we fit a loess curve around the values to get the general trend line? Also, after removing the seasonal component we have some negative values for some reason. Would this affect the loess() curve?
I have a large dataset of over 75.000 observations. Of these observations I have a list of date and time combinations. I want to calculate the observation frequency in a predetermined timestep (15, 30 or 60 minutes). The study period is from 2014-10-21 00:00 to 2015-10-21 23:59.
The raw data is stored in a DF, but date (as POSIXlt) and time (as character) are in different columns, so I combine them back into one column to create a POSIXct timestamp.
receiver$date2 = as.POSIXct(paste(receiver$date, receiver$time), format="%Y-
%m-%d %H:%M:%S")
dateseq = receiver$date2
dateseq is now (only a small fragment using dput()):
dateseq = structure(c(1414140420, 1414140720, 1414140960, 1414141080, 1414143540, 1414144980, 1414145940, 1414147380, 1414147440, 1414148100, 1414148280, 1414152720, 1414153740, 1414154520, 1414154580, 1414158540, 1414159380, 1414159680, 1414164240, 1414164300, 1414164840, 1414164900, 1414165500, 1414166100, 1414166220, 1414166460, 1414166520, 1414166820, 1414166880, 1414166940, 1414167300, 1414167360, 1414167480, 1414167780, 1414168380, 1414168440, 1414168800, 1414168860, 1414202040, 1414202220, 1414202280, 1414202700, 1414202820, 1414202880, 1414203660, 1414203960, 1414215180, 1414215300, 1414215900, 1414216560, 1414216860, 1414217220, 1414217280, 1414217460, 1414217580, 1414217700, 1414217820, 1414217880, 1414218240, 1414218720, 1414219380, 1414219800, 1414219920, 1414219980, 1414220160, 1414220280, 1414220820, 1414220880, 1414221000, 1414221960, 1414222080, 1414222200, 1414222320, 1414222500, 1414222560, 1414222860, 1414223640, 1414224780, 1414225800, 1414225920, 1414225980, 1414226040, 1414226100, 1414226220, 1414227240, 1414227420, 1414227600, 1414230300, 1414230540, 1414230840, 1414231140, 1414231320, 1414231440, 1414231560, 1414231800, 1414231860, 1414232040, 1414232160, 1414232400, 1414232520, 1414232640, 1414232700, 1414232760, 1414232880, 1414232940, 1414233060, 1414233180, 1414233240, 1414233300, 1414233420, 1414233480, 1414233660, 1414233720, 1414233780, 1414233840, 1414233960, 1414234080, 1414234320, 1414234440, 1414234560, 1414234620, 1414234740, 1414234860, 1414234980, 1414235040, 1414235280, 1414236240, 1414236300, 1414236420, 1414236540, 1414236840, 1414236900, 1414236960, 1414237020, 1414237260, 1414237560, 1414237860, 1414238280, 1414238400, 1414238460, 1414238580, 1414238640, 1414239180, 1414239300, 1414239360, 1414239480, 1414239540, 1414240440, 1414240860, 1414240920, 1414240980, 1414241040, 1414242000, 1414242180, 1414242480, 1414242540, 1414242660, 1414242720, 1414242840, 1414242900, 1414243800, 1414243920, 1414244280, 1414244460, 1414245240, 1414245600, 1414245660, 1414246080, 1414246500, 1414246680, 1414246740, 1414246920, 1414247340, 1414248180, 1414249320, 1414249560, 1414249860, 1414250340, 1414250520, 1414250640, 1414250760, 1414250880, 1414250940, 1414251060, 1414251240, 1414251900, 1414252020, 1414252080, 1414252200, 1414252260, 1414252380, 1414252440, 1414252440, 1414252500, 1414252560, 1414252680, 1414252980, 1414253160, 1414253460, 1414253580), class = c("POSIXct", "POSIXt"), tzone = "")
Then I want to have a timeseq that runs for the whole period (so also the days that don't have any observations) divided by the predetermined timestep.
timestep = 1800 # 1800 sec = 30 min
start = "2014-10-21 00:00"
end = "2015-10-21 23:59"
receiver = R125926
timeseq = seq(from = as.POSIXct(start), to = as.POSIXct(end), by = timestep)
Now I want to 'fill' a new dataframe with the timeseq in one column and the count data of how many observations (from dateseq) occurred in that time period.
EDIT
After some searching on the forum and adjusting some code, I came to one very simple method that brings me very close to what I want my results to look like:
det_interval = data.frame(table(cut(dateseq, breaks = "30 min")))
There's only two adjustments that I don't know how to do. Now it begins at the first record (e.g. when my first record is on 05.17 the interval it will use will be x.17 - x.47 (30min)), not at the start that I want (see the timeseq created above). So how can I make sure that this starts and ends at a predetermined date/time?
I'm trying to plot a simple graph that show the increase in wealth for two different investment strategies. When using the standard graph from R it works, but when I try to use ggplot2 I get these weird spikes in the lines.
Does any have any idea what could be causing this?
I've tried to simply the code as much as possible:
For the standard graph
ind.ts = ts(cbind(ind.passive,ind.active), start=c(insample.endstart,1),frequency=12)
plot(log(ind.ts),type="lines", col=c("blue","red"))
legend(x="topleft", legend=c("Passive","Active"), col=c("blue","red"), lty=1)
For the ggplot graph
testers=data.frame(ind.ts)
ggplot(testers, aes(date)) +
geom_line(aes(y = log(ind.passive), colour = "Passive",size="1")) +
geom_line(aes(y = log(ind.active), colour = "Active",size="1"))
The Ind.ts data set
structure(c(1, 1.026669, 1.066102329621, 1.09764083483818, 1.13073909657189,
1.17422279926966, 1.201650295415, 1.24229131005623, 1.24436842112664,
1.29675757602449, 1.29281154272065, 1.34840890311535, 1.37447769243928,
1.42187380670767, 1.43432089001159, 1.44828830683852, 1.47037760009442,
1.50663270057995, 1.51269991046518, 1.44617893190248, 1.47609892782461,
1.55880475075062, 1.60230787373457, 1.72267003659376, 1.6884336922865,
1.7947931958647, 1.80827747714523, 1.73407842742553, 1.83823238001199,
1.94879470474019, 2.03637158997651, 2.19836698633073, 2.07500122615881,
2.18823196806907, 2.11573803119891, 2.21303659177769, 2.25083083069207,
2.27667036862841, 2.44006700098487, 2.56495939036328, 2.59127330874902,
2.54554769994283, 2.64902166839781, 2.62135793511473, 2.24229384954953,
2.38534322797539, 2.58003017155629, 2.73574015247005, 2.89313822640227,
3.01496249083961, 2.92082933195062, 3.03735873897812, 3.15584610338566,
3.08028252428619, 3.25121048184135, 3.15027015001163, 3.13383204036887,
3.04763285626648, 3.24152630621501, 3.30661615444381, 3.5011906754359,
3.32628169286315, 3.26271977599422, 3.58162126961968, 3.47465973202375,
3.4018482373392, 3.48660188432426, 3.43296051433394, 3.64465402445034,
3.45302176049876, 3.43920276741325, 3.16710336206381, 3.18321124976327,
3.29673729577483, 2.9957319937214, 2.80662641161774, 3.02543381329387,
3.04403720581181, 2.97111425050939, 2.94227958670819, 2.75683358891715,
2.53472102032527, 2.58379068455775, 2.78122846592754, 2.80549468429276,
2.76500859050373, 2.71079783207832, 2.81360212906206, 2.64401226073284,
2.62324090041252, 2.43641368348514, 2.24723834303094, 2.26148583412576,
2.01595857860056, 2.19346574740491, 2.32192606890168, 2.18514140418268,
2.12856372294559, 2.09571359900937, 2.1165869064555, 2.29149953181808,
2.41150994529845, 2.44221328992199, 2.48518647497146, 2.53301388868229,
2.50620193667058, 2.64742390960003, 2.6698343529948, 2.80897010046677,
2.86115795596334, 2.89979789415863, 2.85611823847891, 2.81197121886675,
2.84980347964538, 2.90496997540435, 2.80930350417434, 2.81972040156782,
2.85016210302314, 2.89418855702854, 3.00999951213804, 3.11183381563269,
3.03729294841303, 3.09892873421517, 3.04396923311387, 2.98710484387007,
3.08097760069353, 3.08499827646243, 3.20047593194697, 3.16912086924169,
3.19575099190593, 3.14371138275373, 3.25904157854143, 3.26071346687123,
3.3485896948034, 3.35499219829987, 3.3971510302637, 3.44342702159796,
3.34200432210381, 3.3473849490624, 3.36955802696499, 3.4464479715823,
3.53637269205683, 3.65311189099431, 3.71864871831875, 3.7710110109214,
3.82954087282191, 3.75144504580245, 3.79450413203817, 3.96444479409563,
4.09921609487092, 4.03197255405065, 3.90887240000293, 3.96507025849778,
4.11298323942078, 4.18000430130714, 4.00202389816178, 3.973681564915,
3.73688988046171, 3.6132997214452, 3.59812747591486, 3.77562310430174,
3.82238042082541, 3.50029900180582, 3.47233161278139, 3.52122551422096,
3.20811814149644, 2.67119786498117, 2.47785656351383, 2.50381211101664,
2.29590056094204, 2.04999813136234, 2.23149881591877, 2.44744541933286,
2.58359925545577, 2.59022877114527, 2.78828284344458, 2.88774646903593,
2.99667515359443, 2.94310059519847, 3.1174675330616, 3.17829867703423,
3.06610473373492, 3.15882374088307, 3.34981254190434, 3.40448483240076,
3.13064849939144, 2.96722864772321, 3.17659630110655, 3.0311907820197,
3.30193068028814, 3.42901538831107, 3.42659107443153, 3.65581631094671,
3.74411158648869, 1, 1.026669, 1.066102329621, 1.09764083483818,
1.13073909657189, 1.17422279926966, 1.201650295415, 1.24229131005623,
1.24436842112664, 1.29675757602449, 1.29281154272065, 1.34840890311535,
1.37447769243928, 1.42187380670767, 1.43432089001159, 1.44828830683852,
1.47037760009442, 1.50663270057995, 1.51269991046518, 1.44617893190248,
1.47609892782461, 1.55880475075062, 1.60230787373457, 1.72267003659376,
1.6884336922865, 1.7947931958647, 1.80827747714523, 1.73407842742553,
1.83823238001199, 1.94879470474019, 2.03637158997651, 2.19836698633073,
2.07500122615881, 2.18823196806907, 2.11573803119891, 2.21303659177769,
2.25083083069207, 2.27667036862841, 2.44006700098487, 2.56495939036328,
2.59127330874902, 2.54554769994283, 2.64902166839781, 2.62135793511473,
2.24229384954953, 2.2509042579318, 2.25833224198298, 2.39462710945113,
2.53239958556629, 2.63903386731532, 2.55663795191, 2.6586375796394,
2.76235103162114, 2.69620929852, 2.84582464870417, 2.75747033083585,
2.74308185064955, 2.66763064126559, 2.83734797029354, 2.89432191753704,
3.06463539645259, 2.91153540595201, 2.85589887587967, 3.13503728790702,
3.04141253434097, 2.97767973468385, 3.05186564759377, 3.00491269460554,
3.19021063591839, 3.02247255089243, 3.01037661574376, 3.02584995154869,
3.04040428981563, 3.05344762421894, 3.06587515604951, 3.07715757662378,
3.08709679559627, 3.09641982791897, 3.10543040961822, 3.1145293207184,
3.12325000281641, 3.13012115282261, 3.13575537089769, 3.14064714927629,
3.14507546175677, 3.14941566589399, 3.15395082445288, 3.15865021118131,
2.96826256970236, 2.97253686780273, 2.97675787015501, 2.98092533117323,
2.98494958037031, 2.98900911179961, 2.99295460382719, 2.99603734706913,
2.99900342404273, 3.00194244739829, 3.00488435099674, 3.00770894228668,
3.01053618869243, 3.16820398996663, 3.20854156316688, 3.26499906051237,
3.32783396743193, 3.29260884488666, 3.47814406068718, 3.5075865501609,
3.69038091563598, 3.75894450266758, 3.80970904817611, 3.75232340078343,
3.69432373797752, 3.74402716954827, 3.81650404749639, 3.69081893620424,
3.70450449281968, 3.74449832332416, 3.80233958892455, 3.95449020757537,
4.08827852027806, 3.99034789660332, 4.07132402646909, 3.99911909485966,
3.92441155104859, 4.04774010845184, 4.05302240929337, 4.20473514411804,
4.16354135391111, 4.19852759190803, 4.1301587686014, 4.28167777318631,
4.28387427388395, 4.39932468556512, 4.40773619436392, 4.4631238073823,
4.52392047988646, 4.39067292607189, 4.39774190948286, 4.42687255189128,
4.52788935665288, 4.64603104574667, 4.79940117659781, 4.88550243370598,
4.95429519347499, 5.03119080917292, 4.92858973500145, 4.9851600879798,
5.20842546768007, 5.38548589145385, 5.29714238089044, 5.13541532685947,
5.20924719301373, 5.40357295030192, 5.49162417152709, 5.25779630592764,
5.22056059248906, 4.90946738678263, 4.91815714405724, 4.9233212090585,
4.92863839596428, 4.93573563525447, 4.94338602548911, 4.95010903048378,
4.95718768639737, 4.96184744282258, 4.96462607739057, 4.96542041756295,
4.96556938017547, 4.96611559280729, 4.9673571217055, 4.9682512459874,
4.96889711864938, 4.96964245321718, 4.97038789958516, 4.9711334577701,
5.14846373047568, 5.34266893085295, 5.24715269570716, 5.55802550431702,
5.66647925598276, 5.46645253824657, 5.63175806300315, 5.97226541900844,
6.06973876291208, 5.58152539525601, 5.29016976962365, 5.2908574916937,
5.04867378086891, 5.04933010846042, 5.24366872567485, 5.2399614518858,
5.59049391317115, 5.72551552216206), .Dim = c(194L, 2L), .Dimnames = list(
NULL, c("ind.passive", "ind.active")), .Tsp = c(1995, 2011.08333333333,
12), class = c("mts", "ts", "matrix"))
The date data set
structure(c(1995.1, 1995.2, 1995.3, 1995.4, 1995.5, 1995.6, 1995.7,
1995.8, 1995.9, 1995.1, 1995.11, 1995.12, 1996.1, 1996.2, 1996.3,
1996.4, 1996.5, 1996.6, 1996.7, 1996.8, 1996.9, 1996.1, 1996.11,
1996.12, 1997.1, 1997.2, 1997.3, 1997.4, 1997.5, 1997.6, 1997.7,
1997.8, 1997.9, 1997.1, 1997.11, 1997.12, 1998.1, 1998.2, 1998.3,
1998.4, 1998.5, 1998.6, 1998.7, 1998.8, 1998.9, 1998.1, 1998.11,
1998.12, 1999.1, 1999.2, 1999.3, 1999.4, 1999.5, 1999.6, 1999.7,
1999.8, 1999.9, 1999.1, 1999.11, 1999.12, 2000.1, 2000.2, 2000.3,
2000.4, 2000.5, 2000.6, 2000.7, 2000.8, 2000.9, 2000.1, 2000.11,
2000.12, 2001.1, 2001.2, 2001.3, 2001.4, 2001.5, 2001.6, 2001.7,
2001.8, 2001.9, 2001.1, 2001.11, 2001.12, 2002.1, 2002.2, 2002.3,
2002.4, 2002.5, 2002.6, 2002.7, 2002.8, 2002.9, 2002.1, 2002.11,
2002.12, 2003.1, 2003.2, 2003.3, 2003.4, 2003.5, 2003.6, 2003.7,
2003.8, 2003.9, 2003.1, 2003.11, 2003.12, 2004.1, 2004.2, 2004.3,
2004.4, 2004.5, 2004.6, 2004.7, 2004.8, 2004.9, 2004.1, 2004.11,
2004.12, 2005.1, 2005.2, 2005.3, 2005.4, 2005.5, 2005.6, 2005.7,
2005.8, 2005.9, 2005.1, 2005.11, 2005.12, 2006.1, 2006.2, 2006.3,
2006.4, 2006.5, 2006.6, 2006.7, 2006.8, 2006.9, 2006.1, 2006.11,
2006.12, 2007.1, 2007.2, 2007.3, 2007.4, 2007.5, 2007.6, 2007.7,
2007.8, 2007.9, 2007.1, 2007.11, 2007.12, 2008.1, 2008.2, 2008.3,
2008.4, 2008.5, 2008.6, 2008.7, 2008.8, 2008.9, 2008.1, 2008.11,
2008.12, 2009.1, 2009.2, 2009.3, 2009.4, 2009.5, 2009.6, 2009.7,
2009.8, 2009.9, 2009.1, 2009.11, 2009.12, 2010.1, 2010.2, 2010.3,
2010.4, 2010.5, 2010.6, 2010.7, 2010.8, 2010.9, 2010.1, 2010.11,
2010.12, 2011.1, 2011.2), .Tsp = c(1995, 2011.08333333333, 12
), class = "ts")
The spikes are in your data, specifically in the crummy way the dates are stored. January, February, March 1995 are coded as 1995.10, 1995.20, 1995.30, but then October, November, and December are 1995.10, 1995.11, 1995.12. When you pass your time series to ggplot you maybe saw a warning like:
Don't know how to automatically pick scale for object of type ts. Defaulting to continuous
So ggplot just converted to numerics, giving October the same x value as January and inserting Nov and Dec before February, causing your spikes. Since your samples (as far as I checked) are spaced every month, you could add a new column to your data like this:
ind.df <- as.data.frame(ind.ts)
ind.df$date <- seq(as.Date('1995-01-01'), as.Date('2011-02-01'), by = "month")
Then, ggplot works best with long-format data, so we can melt it
library(reshape2)
ind.melt <- melt(ind.df, id.vars = "date")
ggplot(ind.melt, aes(x = date, y = value, color = variable) +
geom_line(size = 1)
And the spikes are gone.
One other note, in ggplot don't put anything inside aes() that isn't mapping to a data column. In your post, inside aes() you have size = "1". You don't need the quotes around 1, and since it applies to the whole layer you should put it outside of aes().
The following example illustrates that for a very simple example, the plots from the basic R plotting and ggplot2 are the same, i.e. basic plotting does not get rid of the spikes, nor does ggplot2 introduces spikes. You need to make your example more complete, i.e. provide us with a sample of your data that reproduces the issue you see.
x = 1:100
y = runif(100)
y[50] = 5
plot(x, y)
library(ggplot2)
qplot(x, y, geom = 'line')