Easy R
Download as PPT, PDF1 like803 views
The document discusses creating an optimized algorithm in R. It covers: 1) Background on R and some popular R packages and interfaces. 2) Optimizing code performance by using parallel computing techniques like multiple cores and high performance computing clusters. 3) Steps for writing functions in R, creating R packages, and optimizing code performance.
![Writing a Function/ Algorithm in R 2 Adding loops and multiple function Eg- # Arrays of values for each type of species and sex species <- unique ( crabs $ sp ) sexes <- unique ( crabs $ sex ) # Loop through species ... for ( i in 1 :length ( species )){ # ... loop through sex .. for ( j in 1 :length ( sexes )){ #... and finally call a function on each subset something_else ( subset ( crabs , sp == species [ i ] & sex == sexes [ j ])) } Citation- http://cran.r-project.org/doc/manuals/R-exts.html#Top http://www.bioinformaticszen.com/r_programming/data_analysis_using_r_functions_as_objects/](https://image.slidesharecdn.com/cfakepatheasyr-091022112603-phpapp02/85/Easy-R-10-320.jpg)
![Writing a Function/ Algorithm in R 2 Adding loops and multiple function Eg- # Arrays of values for each type of species and sex species <- unique ( crabs $ sp ) sexes <- unique ( crabs $ sex ) # Loop through species ... for ( i in 1 :length ( species )){ # ... loop through sex .. for ( j in 1 :length ( sexes )){ #... and finally call a function on each subset something_else ( subset ( crabs , sp == species [ i ] & sex == sexes [ j ])) } Citation- http://cran.r-project.org/doc/manuals/R-exts.html#Top http://www.bioinformaticszen.com/r_programmin/data_analysis_using_r_functions_as_objects/](https://image.slidesharecdn.com/cfakepatheasyr-091022112603-phpapp02/85/Easy-R-11-320.jpg)
![More ways to write functions each <- function ( . column , . data , . lambda ){ # Find the column index from it's name column_index <- which(names( . data) == . column) # Find the unique values in the column column_levels <- unique( . data[,column_index]) # Loop over these values for (i in 1 :length(column_levels)){ # Subset the data and call the passed function on it . lambda( . data[ . data[,column_index] == column_levels[i],]) } } The last argument .lambda is an R function, because R treats functions as objects this allows them to be passed as arguments to other functions. # Another function as the last argument to this function each ( "sp" , crabs , something_else ) # Or create a new anonymous function ... each ( "sp" , crabs , function ( x ){ # ... and run multiple lines of code here something_else ( x ) with ( x , lm ( CW ~ CL )) })](https://image.slidesharecdn.com/cfakepatheasyr-091022112603-phpapp02/85/Easy-R-12-320.jpg)
![Optimizing Code :Example Citation: Dirk Eddelbuettel http://dirk .eddelbuettel.com/papers/ useR 2009 hpcTutorial .pdf > sillysum <- function(N) { s <- 0; for (i in 1:N) s <- s + i; return(s) } > system.time(print(sillysum(1e7))) [1] 5e+13 user system elapsed 13.617 0.020 13.701> > system.time(print(sum(as.numeric(seq(1,1e7))))) [1] 5e+13 user system elapsed 0.224 0.092 0.315> Replacing the loop yielded a gain of a factor of more than 40.](https://image.slidesharecdn.com/cfakepatheasyr-091022112603-phpapp02/85/Easy-R-19-320.jpg)
![Running R Parallel We need a HPC cluster and also Que time in terms of how long we can run our query on the shared resource. Using SNOW A simple example: cl <- makeCluster(4, "MPI") print(clusterCall(cl, function() \ Sys.info()[c("nodename","machine")])) stopCluster(cl) and params <- c("A", "B", "C", "D", "E", "F", "G", "H") cl <- makeCluster( 8 , "MPI") res <- parSapply( cl , params , FUN= function(x) myNEWFunction(x)) will ’unroll’ the parameters params one-each over the function argument given, utilising the cluster cl. In other words, we will be running eight copies of myNEWFunction() at once.](https://image.slidesharecdn.com/cfakepatheasyr-091022112603-phpapp02/85/Easy-R-21-320.jpg)
Easy R
- 1. Creating an Optimized Algorithm in R: Version 1 October 22, 2009
- 2. R : Background Nobody owns it , yet R related products have been created by REvolution Computing (Partnering with Microsoft/Intel) http://www.revolution-computing.com/industry/academic.php SAS (Interface to SAS/IML) http://support.sas.com/rnd/app/studio/Rinterface2.html and SPSS (Interface to SPSS including some use of Python) http://insideout.spss.com/2009/01/13/spss-statistics-and-r/ Blue Reference Inc ( Plugin for MS Office) http://inferenceforr.com/default.aspx and Information Focus ( R GUI for Data Mining) http://www.informationbuilders.com/products/webfocus/predictivemodeling.html
- 3. R Packages CRAN - 1783 Packages in R 2.11 1977 Packages in R 2.9 COST -0 $ BUT a lot of hours. Question: Number of People in the World who know all 1977 R Packages?
- 4. Some uses of R Citation: httP://blog.revolution-computing.com library ( maps ) map ( "state" , interior = FALSE ) map ( "state" , boundary = FALSE , col = "gray" , add = TRUE ) GADM is a spatial database of the location of the world's administrative boundaries the spplot function (from the sp package ). the data for Switzerland, and then plot each canton with a color denoting its primary language: library ( sp ) con <- url ( "http://gadm.org/data/rda/CHE_adm1.RData" ) print ( load ( con )) close ( con ) language <- c ( "german" , "german" , "german" , "german" , "german" , "german" , "french" , "french" , "german" , "german" , "french" , "french" , "german" , "french" , "german" , "german" , "german" , "german" , "german" , "german" , "german" , "italian" , "german" , "french" , "french" , "german" , "german" ) gadm $ language <- as.factor ( language ) col = rainbow ( length ( levels ( gadm $ language ))) spplot ( gadm , "language" , col.regions =c ol , main= "Swiss Language Regions" ) AnthroSpace: Download Global Administrative Areas as RData files
- 5. Seven tips for "surviving" R Keep extensive written notes Find a way to search for R answers Learn to convert complex objects to canonical forms with unclass() Learn how to find and inspect classes and methods for objects Learn how to clear pesky attributes from objects Swallow your pride and learn and use R's many one-line idioms, rather than reinventing the wheel John Mount from Win-Vector LLC :Citation
- 6. Writing a Function/ Algorithm in R Simply enough, newRalgorithm(x) <- function(x) OldAlgorithm(x) Eg- do_something <- function ( x , y ){ # Function code goes here ... } # Subset my data orange_girls <- subset ( crabs , sex == 'F' & sp == 'O' ) # Call my function do_something ( orange_girls $ CW , orange_girls $ C Citation- http://cran.r-project.org/doc/manuals/R-exts.html#Top http://www.bioinformaticszen.com/r_programming/data_analysis_using_r_functions_as_objects/
- 7. Writing a new stats algorithm ( in R /other language) Steps ( Basic Idea)- Journal Review of Study Area Existing Algorithm Study for GAP analysis And add creativity Test and Iterate within community Publish
- 8. Choosing Clustering as the area of interest Clustering works with Big Data. Can work with lots of incomplete column variables when other techniques may not be suitable. Works when data cannot be used for regression models. Groups of clusters can be merged and combined to make new clusters so a case for parallel processing Useful for product marketing, business, medicine and financial
- 9. K Means Clustering using R R> data("planets", package = "HSAUR") R> library("scatterplot3d") R> scatterplot3d(log(planets$mass), log(planets$period), + log(planets$eccen), type = "h", angle = 55, + pch = 16, y.ticklabs = seq(0, 10, by = 2), + y.margin.add = 0.1, scale.y = 0.7)
- 10. Writing a Function/ Algorithm in R 2 Adding loops and multiple function Eg- # Arrays of values for each type of species and sex species <- unique ( crabs $ sp ) sexes <- unique ( crabs $ sex ) # Loop through species ... for ( i in 1 :length ( species )){ # ... loop through sex .. for ( j in 1 :length ( sexes )){ #... and finally call a function on each subset something_else ( subset ( crabs , sp == species [ i ] & sex == sexes [ j ])) } Citation- http://cran.r-project.org/doc/manuals/R-exts.html#Top http://www.bioinformaticszen.com/r_programming/data_analysis_using_r_functions_as_objects/
- 11. Writing a Function/ Algorithm in R 2 Adding loops and multiple function Eg- # Arrays of values for each type of species and sex species <- unique ( crabs $ sp ) sexes <- unique ( crabs $ sex ) # Loop through species ... for ( i in 1 :length ( species )){ # ... loop through sex .. for ( j in 1 :length ( sexes )){ #... and finally call a function on each subset something_else ( subset ( crabs , sp == species [ i ] & sex == sexes [ j ])) } Citation- http://cran.r-project.org/doc/manuals/R-exts.html#Top http://www.bioinformaticszen.com/r_programmin/data_analysis_using_r_functions_as_objects/
- 12. More ways to write functions each <- function ( . column , . data , . lambda ){ # Find the column index from it's name column_index <- which(names( . data) == . column) # Find the unique values in the column column_levels <- unique( . data[,column_index]) # Loop over these values for (i in 1 :length(column_levels)){ # Subset the data and call the passed function on it . lambda( . data[ . data[,column_index] == column_levels[i],]) } } The last argument .lambda is an R function, because R treats functions as objects this allows them to be passed as arguments to other functions. # Another function as the last argument to this function each ( "sp" , crabs , something_else ) # Or create a new anonymous function ... each ( "sp" , crabs , function ( x ){ # ... and run multiple lines of code here something_else ( x ) with ( x , lm ( CW ~ CL )) })
- 13. Additionally create new functions use a Plyr From http://had.co.nz/plyr/ plyr is a set of tools that solves a common set of problems: you need to break a big problem down into manageable pieces, operate on each pieces and then put all the pieces back together. It's already possible to do this with split and the apply functions, but plyr just makes it all a bit easier with: consistent names, arguments and outputs input from and output to data.frames, matrices and lists progress bars to keep track of long running operations built-in error recovery a consistent and useful set of tools for solving the split-apply-combine problem. library ( plyr ) # Three arguments # 1. The dataframe # 2. The name of columns to subset by # 3. The function to call on each subset d_ply ( crabs , . ( sp , sex ), something_else )
- 14. Quick Recap We have an algorithm in mind or create a new alogirthm ( toughest part) ( Eg. http://en.scientificcommons.org/42572415 Genetic K-Means (GKM) or Genetic Regularized Mahalanobis (GARM) distances to compute the initial cluster parameters, with little difference in the final results. This innovation allows our algorithm to find optimal parameter estimates of complex hyperellisoidal clusters. We develop and score the information complexity (ICOMP) criterion of Bozdogan (1994a,b, 2004) as our fitness function to choose the number of clusters present in the data sets ) We created a function in R on it.We can also use to rename Package Functions (like a SAS R Package I created) We now need to create a package so we all 2 million R users may have a chance to use it
- 15. Creating a New Package Citation- http://cran.r-project.org/doc/contrib/Leisch-CreatingPackages.pdf 1. Load all functions and data sets you want in the package into a clean R session, and run package.skeleton(). The objects are sorted into data and functions, skeleton help files are created for them using prompt() and a DESCRIPTION file is created. The function then prints out a list of things for you to do next. This creates the Package within the Current Working Directory > package.skeleton(name="NAME_OF_PACKAGE",code_files="FILENAME.R") Creating directories ... Creating DESCRIPTION ... Creating Read-and-delete-me ... Copying code files ... Making help files ... Done. Further steps are described in './linmod/Read-and-delete-me'. Q WHERE IS MY PACKAGE? A getwd()
- 16. Creating a New Package Citation- http://cran.r-project.org/doc/contrib/Leisch-CreatingPackages.pdf Q What is the best step in making a software- A Documenting HELP FINALLY * Edit the help file skeletons in 'man', possibly combining help files for multiple functions. * Put any C/C++/Fortran code in 'src'. * If you have compiled code, add a .First.lib() function in 'R' to load the shared library. * Run R CMD build to build the package tarball. * Run R CMD check to check the package tarball. Read "Writing R Extensions" for more information. http://cran. r -project.org/doc/manuals/ R -exts.pdf Also see guidelines for CRAN submission
- 17. Next Steps We have New functions and a new Package We now need to optimize the R Package for Performance Using 1) Parallel Computing 2) High Performance Computing 3) Code Optimization
- 18. Optimizing Code Citation: Dirk Eddelbuettel http://dirk .eddelbuettel.com/papers/ useR 2009 hpcTutorial .pdf R already provides the basic tools for performance analysis. the system.time function for simple measurements. the Rprof function for profiling R code. the Rprofmem function for profiling R memory usage. In addition, the profr and proftools package on CRAN can be used to visualize Rprof data. We use tools for creating visual images of how the algorithm is looping- in case we dont know how the algorithm we created looks visually and to avoid multiple calls.
- 19. Optimizing Code :Example Citation: Dirk Eddelbuettel http://dirk .eddelbuettel.com/papers/ useR 2009 hpcTutorial .pdf > sillysum <- function(N) { s <- 0; for (i in 1:N) s <- s + i; return(s) } > system.time(print(sillysum(1e7))) [1] 5e+13 user system elapsed 13.617 0.020 13.701> > system.time(print(sum(as.numeric(seq(1,1e7))))) [1] 5e+13 user system elapsed 0.224 0.092 0.315> Replacing the loop yielded a gain of a factor of more than 40.
- 20. Running R Parallel We need a cluster ( like Newton with 1500 processors run on 2 nd floor SMC ) Several R packages to execute code in parallel: NWS Rmpi snow (using MPI, PVM, NWS or sockets) papply taskPR multicore
- 21. Running R Parallel We need a HPC cluster and also Que time in terms of how long we can run our query on the shared resource. Using SNOW A simple example: cl <- makeCluster(4, "MPI") print(clusterCall(cl, function() \ Sys.info()[c("nodename","machine")])) stopCluster(cl) and params <- c("A", "B", "C", "D", "E", "F", "G", "H") cl <- makeCluster( 8 , "MPI") res <- parSapply( cl , params , FUN= function(x) myNEWFunction(x)) will ’unroll’ the parameters params one-each over the function argument given, utilising the cluster cl. In other words, we will be running eight copies of myNEWFunction() at once.
- 22. Current Status We are writing the algorithm we have selected for optimized use on Newton We will create a Package and release it with a paper once project is over