slope and intercept issue in scatterplot ggplot2 - r

I was drawing a regression line (linear) using mtcars dataset (mpg ~ cyl). I ran a simple linear model using mpg and cyl and executed a summary.
Intercept from the linear model summary does not match with the graphical representation.
I am having hard time understanding what is going on. If I use R's base plotting function to draw the scatterplot, I get the same result as ggplot2. I changed the y-axis scale limit (0, 40) without any success.
Here is my code
data(mtcars)
library(ggplot2)
p <- ggplot(mtcars, aes(x=cyl, y=mpg)) + geom_point(shape=1) # create graph
p + geom_smooth(method = lm, se=FALSE) # add line
lm.car <- lm(mpg ~ cyl) # create linear model
summary(lm.car) # summary
Here is linear model output
> summary(lm.car)
Call:
lm(formula = mpg ~ cyl)
Residuals:
Min 1Q Median 3Q Max
-4.9814 -2.1185 0.2217 1.0717 7.5186
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 37.8846 2.0738 18.27 < 2e-16
cyl -2.8758 0.3224 -8.92 6.11e-10
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
Residual standard error: 3.206 on 30 degrees of freedom
Multiple R-squared: 0.7262, Adjusted R-squared: 0.7171
F-statistic: 79.56 on 1 and 30 DF, p-value: 6.113e-10
Based on following suggestions I used
data(mtcars)
library(ggplot2)
p <- ggplot(mtcars, aes(x=cyl, y=mpg)) + geom_point(shape=1) +
xlim(0, 10)# create graph
p + geom_smooth(method = lm, se=FALSE) # add line
Here is the ouput:

When you do regression analysis Intercept value shows how large will be y value if x value is 0. In case of cyl and mpg Intercept value 37.8846 means that mpg will be 37.8846 if cyl value is O.
On the ggplot2 plot regression line only shows values of cyl from 4 to 6 (as there are no other values).
If you calculate predicted value of mpg for cyl value of 4 you will get 26.38142. That's the value you see on plot.
predict(lm.car,data.frame(cyl=4))
1
26.38142

Related

How do I seperate data in R by category

Say I have the data frame (image below) and I want to split in to two new categories based on region so one would be BC and the other NZ, how do I achieve this? (in R)
data
Here is an example with the mtcars data where we use the transmission variable am to plot different groups in a scatterplot with the ggplot2 package.
We will create a scatterplot with the displacement variable on the x axis and miles per gallon on the y axis. Since cars with larger engine displacement typically consume more gasoline than those with smaller displacement, we expect to see a negative relationship (mpg is higher with low values of displacement) in the chart.
First, we convert am to a factor variable so the legend prints two categories instead of a continuum between 0 and 1. Then we use ggplot() and geom_point() to set the point color based on the value of am.
library(ggplot2)
mtcars$am <- factor(mtcars$am,labels = c("automatic","manual"))
ggplot(mtcars,aes(disp,mpg,group = am)) +
geom_point(aes(color = am))
...and the output:
Separating charts by group with facets
We can use ggplot2 directly to generate separate charts by a grouping variable. In ggplot2 this is known as a facetted chart. We use facet_wrap() to split the data by values of am as follows.
ggplot(mtcars,aes(disp,mpg,group = am)) +
geom_point() +
facet_wrap(mtcars$am,ncol = 2)
...and the output:
Adding regression line and confidence intervals
Given the comments in the original question, we an add a regression line to the plot by using the geom_smooth() function, which defaults to lowess smoothing.
ggplot(mtcars,aes(disp,mpg,group = am)) +
geom_point() +
facet_wrap(mtcars$am,ncol = 2) +
geom_smooth(span = 1)
...and the output:
To use a simple regression instead of lowess smoothing, we use the method = argument in geom_smooth(), and set it to lm.
ggplot(mtcars,aes(disp,mpg,group = am)) +
geom_point() +
facet_wrap(mtcars$am,ncol = 2) +
geom_smooth(method = "lm")
...and the output:
Generate regression models by group
Here we split the data frame by values of am, and use lapply() to generate regression models for each group.
carsList <- split(mtcars,mtcars$am)
lapply(carsList,function(x){
summary(lm(mpg ~ disp,data = x))
})
...and the output:
$automatic
Call:
lm(formula = mpg ~ disp, data = x)
Residuals:
Min 1Q Median 3Q Max
-2.7341 -1.6546 -0.8855 1.6032 5.0764
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 25.157064 1.592922 15.79 1.36e-11 ***
disp -0.027584 0.005146 -5.36 5.19e-05 ***
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
Residual standard error: 2.405 on 17 degrees of freedom
Multiple R-squared: 0.6283, Adjusted R-squared: 0.6064
F-statistic: 28.73 on 1 and 17 DF, p-value: 5.194e-05
$manual
Call:
lm(formula = mpg ~ disp, data = x)
Residuals:
Min 1Q Median 3Q Max
-4.6056 -2.4200 -0.0956 3.1484 5.2315
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 32.86614 1.95033 16.852 3.33e-09 ***
disp -0.05904 0.01174 -5.031 0.000383 ***
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
Residual standard error: 3.545 on 11 degrees of freedom
Multiple R-squared: 0.6971, Adjusted R-squared: 0.6695
F-statistic: 25.31 on 1 and 11 DF, p-value: 0.0003834
NOTE: since this is an example illustrating the code necessary to generate a regression analysis with a split variable, we won't go into the details about whether the data here conforms to modeling assumptions for Ordinary Least Squares regression.
Modeling the groups in one regression model
As I noted in the comments, we can account for the differences between automatic and manual transmissions in one regression model if we specify the am effect as well as an interaction effect between am and disp.
summary(lm(mpg ~ disp + am + am * disp,data=mtcars))
We can demonstrate that this model generates the same predictions as the split model for manual transmissions by generating predictions from each model as follows.
data <- data.frame(am = c(1,1,0),
disp = c(157,248,300))
data$am <- factor(data$am,labels = c("automatic","manual"))
mod1 <- lm(mpg ~ disp + am + am * disp,data=mtcars)
predict(mod1,data)
mod2 <- lm(mpg ~ disp,data = mtcars[mtcars$am == "manual",])
predict(mod2,data[data$am == "manual",])
...and the output:
> data <- data.frame(am = c(1,1,0),
+ disp = c(157,248,300))
> data$am <- factor(data$am,labels = c("automatic","manual"))
> mod1 <- lm(mpg ~ disp + am + am * disp,data=mtcars)
> predict(mod1,data)
1 2 3
23.59711 18.22461 16.88199
> mod2 <- lm(mpg ~ disp,data = mtcars[mtcars$am == "manual",])
> predict(mod2,data[data$am == "manual",])
1 2
23.59711 18.22461
We subset the data prior to predict() for the split model in order to generate predictions only for observations that had manual transmissions. Since the predictions match, we prove that building separate models by transmission type is no different than a fully specified model that includes both the categorical am effect and an interaction effect for am * disp.

Save the result from "summary(lm)" to use in PowerBi

Is it possible to save an summary(lm) object in a format usable in PowerBI?
Lets say the following:
data <- mpg
lm <- lm(hwy ~ displ, data = mpg)
summary(lm)
Output:
Call:
lm(formula = hwy ~ displ, data = mpg)
Residuals:
Min 1Q Median 3Q Max
-7.1039 -2.1646 -0.2242 2.0589 15.0105
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 35.6977 0.7204 49.55 <2e-16 ***
displ -3.5306 0.1945 -18.15 <2e-16 ***
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
Residual standard error: 3.836 on 232 degrees of freedom
Multiple R-squared: 0.5868, Adjusted R-squared: 0.585
F-statistic: 329.5 on 1 and 232 DF, p-value: < 2.2e-16
I would lake to save this information as gglpot2 object, or picture in general, that I can display in Power BI. So that we can use this as a template for a fast regressions inside Power BI. This since Power BI only can display R code that results in a "plot" and not text.
I have tried:
textplot(capture.output(summary(lm)))
But I first got this error:
>install.packages('textplot')
Warning in install.packages :
package ‘textplot’ is not available (for R version 3.5.3)
And unfortunately Power BI doesn't supports textplot().
EDIT: Clarification I'm not looking to plot a regression line nor plane. I'm looking for a way to save the text output from "summary(lm)" as a plot object that I can display in Power BI.
Try something like this:
fit <- lm(hwy ~ displ, data = mpg)
txt = capture.output(print(summary(fit)))
plot(NULL,xlim=c(-1,1),ylim=c(-1,1),xaxt="n",yaxt="n",bty="n",xlab="",ylab="")
text(x=0,y=0,paste(txt,collapse="\n"))
You might need to look at using stringr::str_pad to make the text prettier.. but this should get you something that works.
This is how you put it on ggplot2:
ggplot() + xlim(c(-1,1)) + ylim(c(-1,1)) +
geom_text(aes(x=0,y=0,label=paste(txt,collapse="\n"))) +
theme_void()
This example should be useful for you. To apply here:
plot(hwy ~ displ, data=mpg)
abline(lm1)
For the ggplot2 solution, this works:
ggplot(mpg, aes(displ, hwy)) + geom_point() + geom_smooth(method='lm', formula=y~x)

Why is this regression plot only plotting 2 of the 4 regression coefficients? [duplicate]

This question already has answers here:
plot regression line in R
(3 answers)
Closed 5 years ago.
I have the following set of data: https://archive.ics.uci.edu/ml/datasets/abalone
I am trying to plot a regression for the whole weight against the diameter.
A scatter plot of the data is clearly not a linear function. (I am unable to attach it for some reason.)
Consider a quadratic regression model. I set it up like so:
abalone <- read.csv("abalone.data")
diameter <- abalone$Diameter
diameter2 <- diameter^2
whole <- abalone$Whole.weight
quadraticModel <- lm( whole ~ diameter + diameter2)
This is fine and gives me the following when calling quadraticModel:
Call:
lm(formula = whole ~ diameter + diameter2)
Coefficients:
(Intercept) diameter diameter2
0.3477 -3.3555 10.4968
However, when I plot:
abline(quadraticModel)
I get the following warning:
Warning message:
In abline(quadraticModel) :
only using the first two of 3 regression coefficients
which means that I am getting a straight line plot which isn't what I am aiming for. Can someone please explain to me why this is happening and possible ways around it? I am also having the same issue with cubic plots etc. (They always just plot the first two coefficients.)
You can not use abline to plot polynomial regression fitted. Try this:
x<-sort(diameter)
y<-quadraticModel$fitted.values[order(diameter)]
lines(x, y)
I don't think you're producing a quadratic fit, rather a linear fit using diameter and the squared diameter. Try this instead:
library(stats)
df <- read.csv("abalone.data")
var_names <-
c(
"Sex",
"Length",
"Diameter",
"Height",
"Whole_weight",
"Shucked_weight",
"Viscera_weight",
"Shell_weight",
"Rings"
)
colnames(df) <- var_names
fit <- lm(df$Whole_weight ~ poly(df$Diameter, 2))
summary(fit)
diameter <- df$Diameter
predicted_weight <- predict(fit, data.frame(x = diameter))
plot(diameter, predicted_weight)
> summary(fit)
Call:
lm(formula = df$Whole_weight ~ poly(df$Diameter, 2))
Residuals:
Min 1Q Median 3Q Max
-0.66800 -0.06579 -0.00611 0.04590 0.97396
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 0.828818 0.002054 403.44 <2e-16 ***
poly(df$Diameter, 2)1 29.326043 0.132759 220.90 <2e-16 ***
poly(df$Diameter, 2)2 8.401508 0.132759 63.28 <2e-16 ***
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
Residual standard error: 0.1328 on 4173 degrees of freedom
Multiple R-squared: 0.9268, Adjusted R-squared: 0.9267
F-statistic: 2.64e+04 on 2 and 4173 DF, p-value: < 2.2e-16

R - Regression Analysis for Logarthmic

I perform regression analysis and try to find the best fit model for the dataset diamonds.csv in ggplot2. I use price(response variable) vs carat and I perform linear regression, quadratic, and cubic regression. The line is not the best fit. I realize the logarithmic from excel has the best fitting line. However, I couldn't figure out how to code in R to find the logarithmic fitting line. Anyone can help?
Comparing Price vs Carat
model<-lm(price~carat, data = diamonds)
Model 2 uses the polynomial to compare
model2<-lm(price~carat + I(carat^2), data = diamonds)
use cubic in model3
model3 <- lm(price~carat + I(carat^2) + I(carat^3), data = diamonds)
How can I code the log in R to get same result as excel?
y = 0.4299ln(x) - 2.5495
R² = 0.8468
Thanks!
The result you report from excel y = 0.4299ln(x) - 2.5495 does not contain any polynomial or cubic terms. What are you trying to do? price is very skewed and as with say 'income' it is common practice to take the log from that. This also provides the R2 you are referring to, but very different coefficients for the intercept and carat parameter.
m1 <- lm(log(price) ~ carat, data = diamonds)
summary(m1)
Call:
lm(formula = log(price) ~ carat, data = diamonds)
Residuals:
Min 1Q Median 3Q Max
-6.2844 -0.2449 0.0335 0.2578 1.5642
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 6.215021 0.003348 1856 <2e-16 ***
carat 1.969757 0.003608 546 <2e-16 ***
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
Residual standard error: 0.3972 on 53938 degrees of freedom
Multiple R-squared: 0.8468, Adjusted R-squared: 0.8468
F-statistic: 2.981e+05 on 1 and 53938 DF, p-value: < 2.2e-16

How to plot quadratic regression in R?

The following code generates a qudaratic regression in R.
lm.out3 = lm(listOfDataFrames1$avgTime ~ listOfDataFrames1$betaexit + I(listOfDataFrames1$betaexit^2) + I(listOfDataFrames1$betaexit^3))
summary(lm.out3)
Call:
lm(formula = listOfDataFrames1$avgTime ~ listOfDataFrames1$betaexit +
I(listOfDataFrames1$betaexit^2) + I(listOfDataFrames1$betaexit^3))
Residuals:
Min 1Q Median 3Q Max
-14.168 -2.923 -1.435 2.459 28.429
Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept) 199.41 11.13 17.913 < 2e-16 ***
listOfDataFrames1$betaexit -3982.03 449.49 -8.859 1.14e-12 ***
I(listOfDataFrames1$betaexit^2) 32630.86 5370.27 6.076 7.87e-08 ***
I(listOfDataFrames1$betaexit^3) -93042.90 19521.05 -4.766 1.15e-05 ***
---
Signif. codes: 0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1
Residual standard error: 7.254 on 63 degrees of freedom
Multiple R-squared: 0.9302, Adjusted R-squared: 0.9269
F-statistic: 279.8 on 3 and 63 DF, p-value: < 2.2e-16
But how to do I plot the curve on the graph am confused.
To get graph:
plot(listOfDataFrames1$avgTime~listOfDataFrames1$betaexit)
But curve?
Is there any to do it without manually copying the values?
Like mso suggested though it works.
This should work.
# not tested
lm.out3 = lm(avgTime ~ poly(betaexit,3,raw=TRUE),listofDataFrames3)
plot(avgTime~betaexit,listofDataDFrames3)
curve(predict(lm.out3,newdata=data.frame(betaexit=x)),add=T)
Since you didn't provide any data, here is a working example using the built-in mtcars dataset.
fit <- lm(mpg~poly(wt,3,raw=TRUE),mtcars)
plot(mpg~wt,mtcars)
curve(predict(fit,newdata=data.frame(wt=x)),add=T)
Some notes:
(1) It is a really bad idea to reference external data structures in the formula=... argument to lm(...). Instead, reference columns of a data frame referenced in the data=... argumennt, as above and as #mso points out.
(2) You can specify the formula as #mso suggests, or you can use the poly(...) function with raw=TRUE.
(3) The curve(...) function takes an expression as its first argument, This expression has to have a variable x, which will be populated automatically by values from the x-axis of the graph. So in this example, the expression is:
predict(fit,newdata=data.frame(wt=x))
which uses predict(...) on the model with a dataframe having wt (the predictor variable) given by x.
Try with ggplot:
library(ggplot)
ggplot(listOfDataFrames1, aes(x=betaexit, y=avgTime)) + geom_point()+stat_smooth(se=F)
Using mtcars data:
ggplot(mtcars, aes(x=wt, y=mpg)) + geom_point()+stat_smooth(se=F, method='lm', formula=y~poly(x,3))
Try:
with(listOfDataFrames1, plot(betaexit, avgTime))
with(listOfDataFrames1, lines(betaexit, 199-3982*betaexit+32630*betaexit^2-93042*betaexit^3))

Resources