piecewise regression in r - r

I have two variables, A and B, that are significantly related if modeled in a piecewise regression. The model has two segments. The problem is that in the plot, the two segments do not connect to one another the way they should: they form a 'nose' at the break point. I've seen in other posts on Stackoverflow that problems with plotting segmented regressions correctly seem widespread.
Here's the dataframe with A and B:
dfrm <- read.table(text=" A B
1 0.04545455 1.3
2 0.09090909 1.1
3 0.13636364 1.6
4 0.18181818 1.8
5 0.22727273 3.4
6 0.27272727 1.8
7 0.31818182 1.9
8 0.36363636 0.7
9 0.40909091 2.9
10 0.45454545 1.2
11 0.50000000 0.8
12 0.54545455 0.7
13 0.59090909 0.6
14 0.63636364 1.7
15 0.68181818 0.7
16 0.72727273 2.0
17 0.77272727 1.2
18 0.81818182 0.5
19 0.86363636 2.8
20 0.90909091 1.0
21 0.95454545 0.5
22 1.00000000 1.0
23 0.06666667 0.2
24 0.13333333 0.6
25 0.20000000 1.6
26 0.26666667 0.4
27 0.33333333 1.7
28 0.40000000 2.5
29 0.46666667 0.5
30 0.53333333 1.5
31 0.60000000 0.4
32 0.66666667 0.3
33 0.73333333 0.2
34 0.80000000 0.2
35 0.86666667 0.7
36 0.93333333 2.2
37 1.00000000 2.3
38 0.05882353 1.4
39 0.11764706 2.7
40 0.17647059 0.7
41 0.23529412 0.2
42 0.29411765 0.8
43 0.35294118 2.9
44 0.41176471 0.4
45 0.47058824 0.5
46 0.52941176 2.1
47 0.58823529 0.4
48 0.64705882 0.6
49 0.70588235 1.0
50 0.76470588 0.3
51 0.82352941 0.9
52 0.88235294 1.4
53 0.94117647 0.6
54 1.00000000 0.4
55 0.10000000 1.7
56 0.20000000 1.4
57 0.30000000 1.5
58 0.40000000 0.6
59 0.50000000 0.4
60 0.60000000 0.5
61 0.70000000 0.4
62 0.80000000 1.0
63 0.90000000 0.8
64 1.00000000 3.0
65 0.03846154 1.5
66 0.07692308 2.7
67 0.11538462 2.2
68 0.15384615 0.6
69 0.19230769 0.7
70 0.23076923 0.5
71 0.26923077 0.5
72 0.30769231 0.6
73 0.34615385 1.2
74 0.38461538 0.8
75 0.42307692 1.8
76 0.46153846 2.1
77 0.50000000 0.6
78 0.53846154 0.7
79 0.57692308 1.3
80 0.61538462 0.4
81 0.65384615 0.7
82 0.69230769 1.2
83 0.73076923 0.8
84 0.76923077 1.2
85 0.80769231 1.0
86 0.84615385 1.4
87 0.88461538 0.9
88 0.92307692 0.8
89 0.96153846 1.7
90 1.00000000 5.8", header=TRUE)
## attach(df) NO, don't use attach and mistrust anyone who tells you differently
model <- lm(B ~ (A < 0.89394)*A + (A >= 0.89394)*A, data=dfrm) # 0.89394 = breakpoint
# Preparing the plot:
a <- sort(unique(dfrm$A))
# Plotting:
plot(B ~ A, data=dfrm)
lines(a, predict(model, list(A=a)), lwd=2, col="blue")
This is the plot:Piecewise regression
How can the two segments be connected cleanly at the break point?

It might be easiest to attempt this with a GAM (Generalized Additive Model), applied via either the GAM package or the mgcv package in R. This technique allows you to fit a non-linear model in stages, smoothing out the joins (or 'knots) between functions. As a bonus, the GAM is basically a GLM anyway so the learning curve should be quite easy.

The nose and the disconnect between the segments may be due to lack of precision in the way the break point is determined.
After re-determining the break point for my data based on the method detailed in Crawley (2007: 427), the two segments perfectly connect.
The steps involved are:
define a vector "breaks" for potential breaks
run a for loop for piecewise regressions for all potential break points and yank out the minimal residual standard error (mse) for each model:
mse <- numeric(length(breaks))
for(i in 1:length(breaks)){
piecewise <- lm(V_indep ~ V_dep*(V_dep < breaks[i]) + V_dep*(V_dep>=breaks[i]))
mse[i] <- summary(piecewise)[6]
}
mse <- numeric(length(breaks))
identify the break point with the least mse:
breaks[which(mse==min(mse))]
fit the model using this break point.

Related

Display a function that is seperated in intervals and plot as piecewise constant graph

at first I am sorry, that I can't describe my problem that well, I hope you understand.
What I have is a mathematical function in a graph (picture one), what I want to describe is a process in which I have used that graph.
First I divided the whole thing in intervals, as seen in the second picture. Than I wrote a program that iterated each interval and called the function at the beginning of each interval and returned a rough and a rounded value. The interval frequency is set for an experiment but can be easily adjusted.
Now I got a set of rounded numbers equal the number of intervals that I want to display in an angular graph as seen in the third picture.
I am not sure if this three graphs describe my procedure or if this is a common problem with a simple solution or not.
I use rstudio as a tool to describe that, and i have a bit experience with ggplot2, but I am open minded if you suggest me to use a different library or approach.
Here is some example data for the function (-0.06x^3)+(0.43x^2)-x+3:
myTable <- "ID Data Rounded
1 2.973 3
2 2.976 3
3 2.970 3
4 2.978 3
5 2.976 3
6 2.973 3
7 2.630 2.6
8 2.630 2.6
9 2.633 2.6
10 2.632 2.6
11 2.630 2.6
12 2.273 2.3
13 2.273 2.3
14 2.273 2.3
15 2.273 2.3
16 2.179 2.2
17 2.179 2.2
18 2.179 2.2
19 2.179 2.2
20 2.179 2.2
21 2.179 2.2
22 2.179 2.2
23 2.179 2.2
24 2.179 2.2
25 2.179 2.2
26 2.179 2.2
27 2.179 2.2
28 2.179 2.2
29 2.179 2.2
30 2.179 2.2
31 2.073 2.1
32 2.073 2.1
33 2.073 2.1
34 2.073 2.1
35 2.073 2.1
36 2.073 2.1
37 2.076 2.1
38 2.073 2.1
39 2.073 2.1
40 1.886 1.9
41 1.886 1.9
42 1.886 1.9
43 1.886 1.9
44 1.886 1.9
45 1.628 1.6
46 1.628 1.6
47 1.631 1.6
48 1.628 1.6
49 1.630 1.6
50 1.628 1.6
51 1.631 1.6
52 1.631 1.6
53 1.631 1.6"
Data <- read.table(text=myTable, header = TRUE)
If my understanding is correct, what you want is to plot a piecewise constant function.
In this case, since you are familiar with ggplot2, you can achieve it using geom_step():
ggplot(Data) + geom_step(aes(x = ID, y = Rounded))

Wrong Fit using nls function

When I try to fit an exponential decay and my x axis has decimal number, the fit is never correct. Here's my data below:
exp.decay = data.frame(time,counts)
time counts
1 0.4 4458
2 0.6 2446
3 0.8 1327
4 1.0 814
5 1.2 549
6 1.4 401
7 1.6 266
8 1.8 182
9 2.0 140
10 2.2 109
11 2.4 83
12 2.6 78
13 2.8 57
14 3.0 50
15 3.2 31
16 3.4 22
17 3.6 23
18 3.8 20
19 4.0 19
20 4.2 9
21 4.4 7
22 4.6 4
23 4.8 6
24 5.0 4
25 5.2 6
26 5.4 2
27 5.6 7
28 5.8 2
29 6.0 0
30 6.2 3
31 6.4 1
32 6.6 1
33 6.8 2
34 7.0 1
35 7.2 2
36 7.4 1
37 7.6 1
38 7.8 0
39 8.0 0
40 8.2 0
41 8.4 0
42 8.6 1
43 8.8 0
44 9.0 0
45 9.2 0
46 9.4 1
47 9.6 0
48 9.8 0
49 10.0 1
fit.one.exp <- nls(counts ~ A*exp(-k*time),data=exp.decay, start=c(A=max(counts),k=0.1))
plot(exp.decay, col='darkblue',xlab = 'Track Duration (seconds)',ylab = 'Number of Particles', main = 'Exponential Fit')
lines(predict(fit.one.exp), col = 'red', lty=2, lwd=2)
I always get this weird fit. Seems to me that the fit is not recognizing the right x axis, because when I use a different set of data, with only integers in the x axis (time) the fit works! I don't understand why it's different with different units.
You need one small modification:
lines(predict(fit.one.exp), col = 'red', lty=2, lwd=2)
should be
lines(exp.decay$time, predict(fit.one.exp), col = 'red', lty=2, lwd=2)
This way you make sure to plot against the desired values on your abscissa.
I tested it like this:
data = read.csv('exp_fit_r.csv')
A0 <- max(data$count)
k0 <- 0.1
fit <- nls(data$count ~ A*exp(-k*data$time), start=list(A=A0, k=k0), data=data)
plot(data)
lines(data$time, predict(fit), col='red')
which gives me the following output:
As you can see, the fit describes the actual data very well, it was just a matter of plotting against the correct abscissa values.

Does ggplot2 exclude some data?

I want to create some basic grouped barplots with ggplot2 but it seems to exclude some data. If I review my input data everything is there, but some bars are missing and it is also messing with the error bars. I tried to convert into multiple variable types, regrouped, loaded again, saved everything in .csv and loaded all new... I just don't know what is wrong.
Here is my code:
library(ggplot2)
limits <- aes(ymax = DataCm$mean + DataCm$sd,
ymin = DataCm$mean - DataCm$sd)
p <- ggplot(data = DataCm, aes(x = factor(DataCm$Zeit), y = factor(DataCm$mean)
) )
p + geom_bar(stat = "identity",
position = position_dodge(0.9),fill =DataCm$group) +
geom_errorbar(limits, position = position_dodge(0.9),
width = 0.25) +
labs(x = "Time [min]", y = "Individuals per foodsource")
This is DataCm:
Zeit mean sd group
1 30 0.1 0.3162278 1
2 60 0.0 0.0000000 2
3 90 0.1 0.3162278 3
4 120 0.0 0.0000000 4
5 150 0.1 0.3162278 5
6 180 0.1 0.3162278 6
7 240 0.3 0.6749486 1
8 300 0.3 0.6749486 2
9 360 0.3 0.6749486 3
10 30 0.1 0.3162278 4
11 60 0.1 0.3162278 5
12 90 0.2 0.4216370 6
13 120 0.3 0.4830459 1
14 150 0.3 0.4830459 2
15 180 0.4 0.5163978 3
16 240 0.3 0.4830459 4
17 300 0.4 0.5163978 5
18 360 0.4 0.5163978 6
19 30 1.2 1.1352924 1
20 60 1.8 1.6865481 2
21 90 2.2 2.0976177 3
22 120 2.2 2.0976177 4
23 150 2.0 1.8856181 5
24 180 2.3 1.9465068 6
25 240 2.4 2.0655911 1
26 300 2.1 1.8529256 2
27 360 2.0 2.1602469 3
28 30 0.2 0.4216370 4
29 60 0.1 0.3162278 5
30 90 0.1 0.3162278 6
31 120 0.1 0.3162278 1
32 150 0.0 0.0000000 2
33 180 0.1 0.3162278 3
34 240 0.1 0.3162278 4
35 300 0.1 0.3162278 5
36 360 0.1 0.3162278 6
37 30 1.3 1.5670212 1
38 60 1.5 1.5811388 2
39 90 1.5 1.7159384 3
40 120 1.5 1.9002924 4
41 150 1.9 2.1317703 5
42 180 1.9 2.1317703 6
43 240 2.2 2.3475756 1
44 300 2.4 2.3190036 2
45 360 2.2 2.1499354 3
46 30 2.1 2.1317703 4
47 60 3.0 2.2110832 5
48 90 3.3 2.1628171 6
49 120 3.2 2.1499354 1
50 150 3.4 2.6331224 2
51 180 3.5 2.4152295 3
52 240 3.7 2.6267851 4
53 300 3.7 2.4060110 5
54 360 3.8 2.6583203 6
The output is:
Maybe you can help me. Thanks in advance!
Best wishes,
Benjamin
Solved it:
I reshaped everything in Excel and exported it another way. The group variable was also not the way I wanted it. Now it is fixed, but I can't really tell you why.
Your data looks malformed. I guess you wanted to have 6 different group values for each time point, but now the group variable just loops over, and you have:
1 30 0.1 0.3162278 1
...
10 30 0.1 0.3162278 4
...
19 30 1.2 1.1352924 1
...
28 30 0.2 0.4216370 4
geom_bar then probably omits rows that have identical mean and time. Although I am not sure why it chooses to do so, you should solve the group problem first anyway.

comparing recent averaged values to a current value in R

I am using Rstudio (version .99.903), have a PC (windows 8). I have a follow up question from yesterday as the problem became more complicated. Here is what the data looks like:
Number Trial ID Open date Enrollment rate
420 NCT00091442 9 1/28/2005 0.2
1476 NCT00301457 26 2/22/2008 1
10559 NCT01307397 34 7/28/2011 0.6
6794 NCT00948675 53 5/12/2010 0
6451 NCT00917384 53 8/17/2010 0.3
8754 NCT01168973 53 1/19/2011 0.2
8578 NCT01140347 53 12/30/2011 2.4
11655 NCT01358877 53 4/2/2012 0.3
428 NCT00091442 55 9/7/2005 0.1
112 NCT00065325 62 10/15/2003 0.2
477 NCT00091442 62 11/11/2005 0.1
16277 NCT01843374 62 12/16/2013 0.2
17386 NCT01905657 62 1/8/2014 0.6
411 NCT00091442 66 1/12/2005 0
What I need to do is compare the enrollment rate of the most current date within a given ID to the average of those values that are up to one year prior to it. For instance, for ID 53, the date of 1/19/2011 has an enrollment rate of 0.2 and I would want to compare this against the average of 8/17/2010 and 5/12/2010 enrollment rates (e.g., 0.15).
If there are no other dates within the ID prior to the current one, then the comparison should not be made. For instance, for ID 26, there would be no comparison. Similarly, for ID 53, there would be no comparison for 5/12/2010.
When I say "compare" I am not doing any analysis or visualization. I simply want a new column that takes the average value of those enrollment rates up to one year prior to the current one (I will be plotting them and percentile ranking them later). There are >20,000 data points. Any help would be much appreciated.
Verbose but possibly high performance way of doing this. No giant for loops looping over all the rows of the data frame. The two sapply loops only operate on a big numeric vector, which should be relatively quick regardless of your data row count. But I'm sure someone will waltz in with a trivial dplyr solution soon enough.
Approach assumes that your data is first sorted by ID then by Opendata. If they are not sorted, you need to sort them first.
# Find indices where the same ID is above and below it
A = which(unlist(sapply(X = rle(df$ID)$lengths,
FUN = function(x) {if(x == 1) return(F)
if(x == 2) return(c(F,F))
if(x >= 3) return(c(F,rep(T, x-2),F))})))
# Store list of date, should speed up code a tiny bit
V_opendate = df$Opendate
# Further filter on A, where the date difference < 365 days
B = A[sapply(A, function(x) (abs(V_opendate[x]-V_opendate[x-1]) < 365) & (abs(V_opendate[x]-V_opendate[x+1]) < 365))]
# Return actual indices of rows - 1, rows +1
C = sapply(B, function(x) c(x-1, x+1), simplify = F)
# Actually take the mean of these cases
D = sapply(C, function(x) mean(df[x,]$Enrollment))
# Create new column rate and fill in with value of C. You can do the comparison from here.
df[B,"Rate"] = D
Number Trial ID Opendate Enrollmentrate Rate
1 420 NCT00091442 9 2005-01-28 0.2 NA
2 1476 NCT00301457 26 2008-02-22 1.0 NA
3 10559 NCT01307397 34 2011-07-28 0.6 NA
4 6794 NCT00948675 53 2010-05-12 0.0 NA
5 6451 NCT00917384 53 2010-08-17 0.3 0.10
6 8754 NCT01168973 53 2011-01-19 0.2 1.35
7 8578 NCT01140347 53 2011-12-30 2.4 0.25
8 11655 NCT01358877 53 2012-04-02 0.3 NA
9 428 NCT00091442 55 2005-09-07 0.1 NA
10 112 NCT00065325 62 2003-10-15 0.2 NA
11 477 NCT00091442 62 2005-11-11 0.1 NA
12 16277 NCT01843374 62 2013-12-16 0.2 NA
13 17386 NCT01905657 62 2014-01-08 0.6 NA
14 411 NCT00091442 66 2005-01-12 0.0 NA
14 411 NCT00091442 66 1/12/2005 0.00 NA
The relevant rows are calculated. You can do your comparison with the newly created Rate column.
You might have to change the code a little since I changed removed the space in the column names
df = read.table(text = " Number Trial ID Opendate Enrollmentrate
420 NCT00091442 9 1/28/2005 0.2
1476 NCT00301457 26 2/22/2008 1
10559 NCT01307397 34 7/28/2011 0.6
6794 NCT00948675 53 5/12/2010 0
6451 NCT00917384 53 8/17/2010 0.3
8754 NCT01168973 53 1/19/2011 0.2
8578 NCT01140347 53 12/30/2011 2.4
11655 NCT01358877 53 4/2/2012 0.3
428 NCT00091442 55 9/7/2005 0.1
112 NCT00065325 62 10/15/2003 0.2
477 NCT00091442 62 11/11/2005 0.1
16277 NCT01843374 62 12/16/2013 0.2
17386 NCT01905657 62 1/8/2014 0.6
411 NCT00091442 66 1/12/2005 0", header = T)

Why length function does not work correct in R?

Following R code gives the cars which are in Type Small. But length function returns 6 instead of 13. Why is that?
> fuel.frame[fuel.frame$Type=="Small",]
row.names Weight Disp. Mileage Fuel Type
1 Eagle.Summit.4 30 0.97 33 3.030303 Small
2 Ford.Escort.4 28 114.00 33 3.030303 Small
3 Ford.Festiva.4 23 0.81 37 2.702703 Small
4 Honda.Civic.4 27 0.91 32 3.125000 Small
5 Mazda.Protege.4 29 113.00 32 3.125000 Small
6 Mercury.Tracer.4 27 0.97 26 3.846154 Small
7 Nissan.Sentra.4 27 0.97 33 3.030303 Small
8 Pontiac.LeMans.4 28 0.98 28 3.571429 Small
9 Subaru.Loyale.4 27 109.00 25 4.000000 Small
10 Subaru.Justy.3 24 0.73 34 2.941176 Small
11 Toyota.Corolla.4 28 0.97 29 3.448276 Small
12 Toyota.Tercel.4 25 0.89 35 2.857143 Small
13 Volkswagen.Jetta.4 28 109.00 26 3.846154 Small
> length(fuel.frame[fuel.frame$Type=="Small",])
[1] 6
length gives in this case the number of columns in the data frame. You can instead use nrow or ncol to get the number of rows or number of columns respectively:
nrow(fuel.frame[fuel.frame$Type=="Small",])
Another example using iris dataset:
> d = head(iris)
> d
Sepal.Length Sepal.Width Petal.Length Petal.Width Species
1 5.1 3.5 1.4 0.2 setosa
2 4.9 3.0 1.4 0.2 setosa
3 4.7 3.2 1.3 0.2 setosa
4 4.6 3.1 1.5 0.2 setosa
5 5.0 3.6 1.4 0.2 setosa
6 5.4 3.9 1.7 0.4 setosa
> nrow(d)
[1] 6
> ncol(d)
[1] 5
> dim(d)
[1] 6 5
I thought it might help to give a bit of an explanation as to thy your getting your result. Your asking the length of the data.frame not the vector. Since the data.frame has 6 columns that explains your result.
this asks for the vector specifically:
length(fuel.frame$Type[fuel.frame$Type=="Small"])
and so does this:
length(fuel.frame[fuel.frame$Type=="Small",][,1])
or use nrow instead of length as already suggested.

Resources