Linear regression with R 2
6 likes2,098 views
This document discusses various methods for model selection in linear regression, including backward elimination, forward selection, stepwise selection, and all-subset regression using the leaps package. It provides examples of using the step(), drop1(), and add1() functions in R, and generating and interpreting results from the regsubsets() function to evaluate all possible regression models and identify the best model based on criteria like adjusted R-squared and Mallow's Cp.
Linear regression with R 2
- 1. Linear Regression with 2: Model selection 2012-12-10 @HSPH Kazuki Yoshida, M.D. MPH-CLE student FREEDOM TO KNOW
- 2. Group Website is at: http://rpubs.com/kaz_yos/useR_at_HSPH
- 3. Previously in this group n Introduction n Graphics n Reading Data into R (1) n Groupwise, continuous n Reading Data into R (2) n Linear regression n Descriptive, continuous n Descriptive, categorical n Deducer
- 4. Menu n Linear regression: Model selection
- 5. Ingredients Statistics Programming n Selection methods n step() n drop1() n add1() n leaps::regsubsets()
- 7. Open the saved script that we created last time. See also Linear Regression with R 1 slides
- 8. Create full & null models lm.full <- lm(bwt ~ age + lwt + smoke + ht + ui + ftv.cat + race.cat + preterm, data = lbw) lm.null <- lm(bwt ~ 1, data = lbw) Intercept-only
- 9. Compare two models anova(lm.full, lm.null) Model 1 Model 2
- 10. Models Partial F-test Difference in residual SS Residual sum of squares Residual degree of freedom Significant
- 11. Backward elimination Specify full model lm.step.bw <- step(lm.full, direction = "backward") Final model object
- 12. Initial AIC Removing ftv.cat for full makes AIC smallest model Removing age makes AIC smallest Doing nothing makes AIC smallest
- 13. Forward selection Final model object Specify null model lm.step.fw <- step(lm.null, scope = ~ age + lwt + smoke + ht + ui + ftv.cat + race.cat + preterm, direction = "forward") formula for possible variables
- 14. Initial AIC for Adding ui null makes AIC smallest model Adding race.cat makes AIC smallest Adding smoke makes AIC smallest Still goes on ...
- 15. Stepwise selection/elimination Final model object Specify null model lm.step.both <- step(lm.null, scope = ~ age + lwt + smoke + ht + ui + ftv.cat + race.cat + preterm, direction = "both") formula for possible variables
- 16. Initial AIC Adding ui for makes AIC smallest null model Adding race.cat Removing is makes AIC smallest also considered Adding smoke Removing is makes AIC smallest also considered Still goes on ...
- 17. F-test using drop1() ## age is the least significant by partial F test drop1(lm.full, test = "F") ## After elimination, ftv.cat is the least significant drop1(update(lm.full, ~ . -age), test = "F") ## After elimination, preterm is least significat at p = 0.12. drop1(update(lm.full, ~ . -age -ftv.cat), test = "F") ## After elimination, all variables are significant at p < 0.1 drop1(update(lm.full, ~ . -age -ftv.cat -preterm), test = "F") ## Show summary for final model summary(update(lm.full, ~ . -age -ftv.cat -preterm))
- 18. Updating models ## Remove age from full model lm.age.less <- update(lm.full, ~ . -age) all variables(.) minus age ## Adding ui to null model lm.ui.only <- update(lm.null, ~ . +ui) all variables (.) plus ui
- 19. test full model age least significant F-test comparing age-in model to age-out model remove age, and test ftv.cat least significant remove age, ftv.cat
- 20. F-test using add1() ## ui is the most significant variable add1(lm.null, scope = ~ age + lwt + race.cat + smoke + preterm + + ui + ftv.cat, test = "F") ## After inclusion, race.cat is the most significant add1(update(lm.null, ~ . +ui), scope = ~ age + lwt + race.cat + smoke + preterm + ht + ui + ftv.cat, test = "F") ## After inclusion, smoke is the most significant add1(update(lm.null, ~ . +ui +race.cat), scope = ~ age + lwt + race.cat + smoke + preterm + ht + ui + ftv.cat, test = "F") ## After inclusion, ht is the most significant add1(update(lm.null, ~ . +ui +race.cat +smoke), scope = ~ age + l + race.cat + smoke + preterm + ht + ui + ftv.cat, test = "F") ...
- 21. test null model ui most significant F-test comparing ui-out model to ui-in model add ui, and test race.cat most significant add ui and race.cat
- 22. All-subset regression using leaps package
- 23. library(leaps) regsubsets.out <- regsubsets(bwt ~ age + lwt + smoke + ht + ui + ftv.cat + race.cat + preterm, data = lbw, nbest = 1, nvmax = NULL, force.in = NULL, force.out = NULL, method = "exhaustive") summary(regsubsets.out)
- 24. library(leaps) Result object regsubsets.out <- regsubsets(bwt ~ age + lwt + smoke + ht + ui + ftv.cat + race.cat + preterm, Full model data = lbw, How many best models? Max model size nbest = 1, Forced variables nvmax = NULL, force.in = NULL, force.out = NULL, method = "exhaustive") summary(regsubsets.out)
- 25. Forced variables Variable combination Best 1 predictor model Best 7 predictor model Best 10 predictor model
- 26. plot(regsubsets.out, scale = "adjr2", main = "Adjusted R^2") the higher the better ~ lwt + smoke + ht + ui + race.cat + preterm ~ smoke + ht + ui + race ~ ui
- 27. library(car) subsets(regsubsets.out, statistic="adjr2", legend = FALSE, min.size = 5, main = "Adjusted R^2") ~ lwt + smoke + ht + ui + race.cat + preterm
- 28. subsets(regsubsets.out, statistic="cp", legend = FALSE, min.size = 5, main = "Mallow Cp") First model for which Mallow Cp is less than number of regressors + 1 ~ lwt + smoke + ht + ui + race.cat + preterm