microsoft r server for distributed computing
7 likes2,131 views
The document introduces Microsoft R Server and Microsoft R Open. It discusses that R is a popular open source programming language and platform for statistics, analytics, and data science. Microsoft R Server allows for distributed computing on big data using R and brings enterprise-grade support and capabilities to the open source R platform. It can perform analytics both in-database using SQL Server and in Hadoop environments without moving data.
microsoft r server for distributed computing
- 1. Introducing Microsoft R Server & Microsoft R Open Krit Kamtuo Technical Evangelist Microsoft (Thailand) Limited
- 2. What is R? Language Platform Community Ecosystem âĒ A programming language for statistics, analytics, and data science âĒ A data visualization framework âĒ Provided as Open Source âĒ Used by 2.5M+ data scientists, statisticians and analysts âĒ Taught in most university statistics programs âĒ Active and thriving user groups across the world âĒ CRAN: 7000+ freely available algorithms, test data and evaluation âĒ Many of these are applicable to big data if scaled âĒ New and recent graduates prefer it
- 3. 20152009200420032000199719951993 Research Projectin New Zealand Open Source Project R-Core Group R-1.0.0 released R Foundation First user New York Times article R-3.2.0 and R Consortium (foundedby Microsoft) History of R
- 4. $? Challenges posed by open source R Uncertain total cost of ownership Inadequate access to important business data Limited business agility Limited business value
- 5. R from Microsoft brings
- 6. 6 âĒ Free and open source R distribution âĒ Enhanced and distributed by Revolution Analytics Microsoft R Open âĒ Built in Advanced Analytics and Stand Alone Server Capability âĒ Leverages the Benefits of SQL 2016 Enterprise Edition SQL Server R Services Microsoft R Products
- 7. Microsoft R Server âĒ Microsoft R Server for Redhat Linux âĒ Microsoft R Server for SUSE Linux âĒ Microsoft R Server for Teradata DB âĒ Microsoft R Server for Hadoop on Redhat Microsoft R Server
- 8. Introducing SQL Server 2016 R Services Enterprise speed and performance Near-DB analytics Parallel threading and processing Model on-premises, store in cloudâor vice versa Hybrid memory and disk scalability Not bound by memory- enabling limits of larger datasets Included in SQL Server 2016 Reuse and optimize existing R code Eliminate data movement across machines Write once, deploy anywhere
- 9. Microsoft R server for distributed computing The First NIDA Business Analytics and Data Sciences Contest/Conference āļ§āļąāļāļāļĩāđ 1-2 āļāļąāļāļĒāļēāļĒāļ 2559 āļ āļāļēāļāļēāļĢāļāļ§āļĄāļīāļāļāļĢāļēāļāļīāļĢāļēāļ āļŠāļāļēāļāļąāļāļāļąāļāļāļīāļāļāļąāļāļāļāļĢāļīāļŦāļēāļĢāļĻāļēāļŠāļāļĢāđ -āđāļāļ°āļāđāļē Microsoft R Server -Distributed Computing āļĄāļĩāļ§āļīāļāļĩāļāļēāļĢāļāļĒāđāļēāļāđāļĢ āđāļĨāļ°āļĄāļĩāļāļĢāļ°āđāļĒāļāļāđāļāļĒāđāļēāļāđāļĢ -āđāļāļ°āļāđāļēāļ§āļīāļāļĩāļāļēāļĢ Configuration āļŠāđāļēāļŦāļĢāļąāļ Distributed Computing https://businessanalyticsnida.wordpress.com https://www.facebook.com/BusinessAnalyticsNIDA/ āļāļĪāļĐāļāļīāđ āļāđāļēāļāļ·āđāļ, Technical Evangelist, Microsoft (Thailand) -Distributed computing āļāļąāļ Big Data -Analytics āļāļ R server -āļŠāļēāļāļīāļāđāļĨāļ°āļŠāļāļāđāļāļĨāļąāļāļĐāļāļ° workshop Computer Lab 2 āļāļąāđāļ 10 āļāļēāļāļēāļĢāļŠāļĒāļēāļĄāļāļĢāļĄāļĢāļēāļāļāļļāļĄāļēāļĢāļĩ 1 āļāļąāļāļĒāļēāļĒāļ 2559 āđāļ§āļĨāļē 9.00-12.30
- 10. Scalable in-database analytics Data Scientist Interacts directly with data Creates models and experiments Data Analyst/DBA Manages data and analytics together Example Solutions âĒ Fraud detection âĒ Sales forecasting âĒ Warehouse efficiency âĒ Predictive maintenance 010010 100100 010101 Relational Data Extensibility ? R R Integration Analytic Library Open Source R Revolution PEMA T-SQL Interface How is it Integrated? âĒ T-SQL calls a Stored Procedure âĒ Script is run in SQL through extensibility model âĒ Result sets sent through Web API to database or applications Benefits âĒ Faster deployment of ML models âĒ Less data movement, faster insights âĒ Work with large datasets: mitigate R memory and scalability limitations
- 11. Cost effectiveness âĒ Best Advanced Analytics Value âĒ R Services and Polybase are built-in o Part of SQL Server 2016 Enterprise Edition âĒ In DB analytics shrinks analysis cost and time o No data movement reduces costs âĒ No Proprietary Hardware Requirement o Can be installed in commodity hardware âĒ Integration between cloud and open source offerings SQL SERVER 2016 $ 648 K + $120 Per user for PowerBI Costs based on a Server with 2 proc/ 8 Cores
- 12. 11
- 13. High-performance open source R plus: âĒ Data source connectivity to big-data objects âĒ Big-data advanced analytics âĒ Multi-platform environment support âĒ In-Hadoop and in-Teradata predictive modeling âĒ Development and production environment support âĒ IDE for data scientist developers âĒ Secure, Scalable R Deployment DeployR R Open R Server DevelopR Microsoft R Server is a broadly deployable enterprise-class analytics platform based on R that is supported, scalable and secure. Supporting a variety of big data statistics, predictive modeling and machine learning capabilities, R Server supports the full range of analytics â exploration, analysis, visualization and modeling Introducing Microsoft R Server
- 14. R Open MicrosoftR Server DeployRDevelopR The Microsoft R Server Platform ConnectR âĒ High-speed & direct connectors Available for: âĒ High-performance XDF âĒ SAS, SPSS, delimited& fixed format text data files âĒ Hadoop HDFS (text & XDF) âĒ Teradata Database & Aster âĒ EDWs and ADWs âĒ ODBC ScaleR âĒ Ready-to-Use high-performance big data big analytics âĒ Fully-parallelizedanalytics âĒ Data prep & data distillation âĒ Descriptive statistics & statistical tests âĒ Range of predictive functions âĒ User tools for distributingcustomizedR algorithms across nodes âĒ Wide data sets supported â thousands of variables DistributedR âĒ Distributed computingframework âĒ Delivers cross-platformportability R+CRAN âĒ Open source R interpreter âĒ R 3.1.2 âĒ Freely-available huge range of R algorithms âĒ Algorithms callable by RevoR âĒ Embeddable in R scripts âĒ 100% Compatible with existingR scripts, functions and packages RevoR âĒ Performance enhancedR interpreter âĒ Based on open source R âĒ Adds high-performance math libraryto speed up linear algebra functions
- 15. ScaleR â Parallel + âBig Dataâ Stream data in to RAM in blocks. âBig Dataâ can be any data size. We handle Megabytes to Gigabytes to TerabytesâĶ Our ScaleR algorithms work inside multiple cores / nodes in parallel at high speed Interim results are collected and combined analytically to produce the output on the entire data set XDF file format is optimised to work with the ScaleR library and significantly speeds up iterative algorithm processing.
- 16. 16
- 17. SQL Server 2016 Enterprise Edition SQL Server R Services Integration Facilities: âĒ Component Integration âĒ Launchers âĒ Parameter Passing âĒ Results Return âĒ Console Output Return âĒ Parallel Data Exchange (RTM) âĒ Stored Procedures âĒ Package Administration SQL Server Query Processor Algorithm Library âĒ Data Prep âĒ Descriptive Stats âĒ Sampling âĒ Statistical Tests âĒ Predictive Models âĒ Variable Selection âĒ Clustering âĒ Classification âĒ Custom APIs for R + CRAN âĒ Parallel Scoring Fast, Parallel, Storage Efficient Algorithms Microsoft R Open âĒ 100% Open Source R âĒ Fully CRAN Compatible âĒ Accelerated Math Open Source R Interpreter
- 18. Run R In-Database from TSQL SQL Server 2016 In-Database Execution of R + CRAN + SQL In-Database Execution of: ï§ R Code ï§ CRAN Packages Move the Work to the Data Run R From the Query Processor Retrieve Models, Scores, Transformed Data, Plots/Images Operationalise scoring/predictio n in database for data batches or real-time
- 19. SQL In-Database Execution: ï§ Remote Execution ï§ Parallelized Compute SQL Server Remote Execution Context Explore and Model: ï§ In Parallel, In-Database ï§ Parallelize distributable R and CRAN Operationlize: ï§ Score In Parallel Parallel Worker Tasks Move BIG Work to the Data Large Data Sets in Chunks Parallel Algorithm Iterate/ Sequence Run Parallel Algorithms in Database from an R client
- 20. SQL 2016 ScaleR PEMAs: Fast, Parallel, Storage Efficient Algorithms R Interpreter Conceptual Flow
- 21. SQL Processor Data Segments (CTP3 is via files) R IDE XSP RTerm.exe R.dll (MSLP$ SQL16) BxlServer.exe (MSLP$SQL16) Input Data Set via ODBC ScaleR Master Process Worker Process Worker Process Worker Process Data Segments Console Out Spawn Worker Procâs. Assemble Intermediate Results Iterate/ Sequence MPI Ring Results â Models, Data Parallelized Algorithms in Database
- 22. 22
- 23. Introducing Microsoft R Server
- 24. ï§ Gradient Boosted Decision Trees ï§ NaÃŊve Bayes Scale R â ParallelizedAlgorithms& Functions ï§ Data import â Delimited, Fixed, SAS, SPSS, OBDC ï§ Variable creation & transformation ï§ Recode variables ï§ Factor variables ï§ Missing value handling ï§ Sort, Merge, Split ï§ Aggregate by category (means, sums) ï§ Min / Max, Mean, Median (approx.) ï§ Quantiles (approx.) ï§ Standard Deviation ï§ Variance ï§ Correlation ï§ Covariance ï§ Sum of Squares (cross product matrix for set variables) ï§ Pairwise Cross tabs ï§ Risk Ratio & Odds Ratio ï§ Cross-Tabulation of Data (standard tables & long form) ï§ Marginal Summaries of Cross Tabulations ï§ Chi Square Test ï§ Kendall Rank Correlation ï§ Fisherâs Exact Test ï§ Studentâs t-Test ï§ Subsample (observations & variables) ï§ Random Sampling Data Preparation Statistical Tests Sampling Descriptive Statistics ï§ Sum of Squares (cross product matrix for set variables) ï§ Multiple Linear Regression ï§ Generalized Linear Models (GLM) exponential family distributions: binomial, Gaussian, inverse Gaussian, Poisson, Tweedie. Standard link functions: cauchit, identity, log, logit, probit. User defined distributions & link functions. ï§ Covariance & Correlation Matrices ï§ Logistic Regression ï§ Classification & Regression Trees ï§ Predictions/scoring for models ï§ Residuals for all models Predictive Models ï§ K-Means ï§ Decision Trees ï§ Decision Forests Cluster Analysis Classification Simulation Variable Selection ï§ Stepwise Regression ï§ Simulation (e.g. Monte Carlo) ï§ Parallel Random Number Generation Combination ï§ rxDataStep ï§ rxExec New ï§ PEMA-R API Custom Algorithms
- 25. ScaleR - Performance comparison Microsoft R Server has no data size limits in relation to size of available RAM. When open source R operates on data sets that exceed RAM it will fail. In contrast Microsoft R Server scales linearly well beyond RAM limits and parallel algorithms are much faster. ï§ US flight data for 20 years ï§ Linear Regression on Arrival Delay ï§ Run on 4 core laptop, 16GB RAM and 500GB SSD
- 26. DistributedR ScaleR ConnectR DevelopR DistributedR - Model development and model compute choice: âWrite Once. DeployAnywhere.â Code Portability Across Platforms In the Cloud Workstations & Servers Linux Windows EDW Teradata Hadoop Hortonworks Cloudera MapR + HD Insights + Hadoop Spark + R Tools for Visual Studio + Azure ML Roadmap Azure Marketplace + SQL Server v16 MicrosoftRServer
- 27. DistributedR - How Does RemoteExecutionWork? Algorithm Master Big Data Predictive Algorithm Analyze Blocks In Parallel Load Block At A Time Distribute Work, Compile Results The Results: âĒ Even Faster Computation âĒ Larger Data Set Capacity âĒ Fewer Security Concerns âĒ No Data Movement, No Copies Work âPack and Shipâ Requests to Remote Environments Results Microsoft R Server functions âĒ A compute context defines remote connection âĒ Microsoft R functions prefixed with rx âĒ Current compute context determines processing location
- 28. DistributedR - Revolution Code Portability ### SETUP HADOOP ENVIRONMENT VARIABLES ### myHadoopCCC <- RxHadoopMR() ### HADOOP COMPUTE CONTEXT ### rxSetComputeContext(myHadoopCC) ### CREATE HDFS, DIRECTORY AND FILE OBJECTS ### hdfsFS <- RxHdfsFileSystem() AirlineDataSet <- RxXdfData(âAirlineDemoSmall/AirlineDemoSmall.xdfâ) , fileSystem = hdfsFS) ### ANALYTICAL PROCESSING ### ### Statistical Summary of the data rxSummary(~ArrDelay+DayOfWeek, data= AirlineDataSet, reportProgress=1) ### CrossTab the data rxCrossTabs(ArrDelay ~ DayOfWeek, data= AirlineDataSet, means=T) ### Linear Model and plot hdfsXdfArrLateLinMod <- rxLinMod(ArrDelay ~ DayOfWeek + 0 , data = AirlineDataSet) plot(hdfsXdfArrLateLinMod$coefficients) ### SETUP LOCAL ENVIRONMENT VARIABLES ### myLocalCC <- âlocalparâ ### LOCAL COMPUTE CONTEXT ### rxSetComputeContext(myLocalCC) ### CREATE LINUX, DIRECTORY AND FILE OBJECTS ### linuxFS <- RxNativeFileSystem() ) AirlineDataSet <- RxXdfData(âAirlineDemoSmall/AirlineDemoSmall.xdfâ, fileSystem = linuxFS) Local Parallel processing â Linux or Windows In â Hadoop ScaleR models can be deployed from a server or edge node to run in Hadoop without any functional R model re-coding for map-reduce Compute context R script â sets where the model will run Functional model R script â does not need to change to run in Hadoop
- 29. DistributedR - In-Hadoop Uses Hadoop nodes for R computations Eliminate data movement latency on very large data Remove data duplication Faster model development No MapReduce R coding Develop better models using all the data = Microsoft R Server
- 30. MRS and Hadoop Architecture options R R R R R R R R R R ScaleR Production RStudio Server Pro Microsoft R Server 1. Copy 2. Stream 3. Send
- 31. DistributedR - Hadoop ProcessingMethods Method 1: Local (Linux) parallel processing using all cores on one node, copying data from HDFS to store in local Linux file-system. Compute Context HadoopCompute Context HadoopCompute Context Local Parallel Linux (Local) File-System HDFS Csv, Xdf Processing Data 1 Edge node 1:n data nodes 1:n disks 1:(n x number of nodes) disks Csv, Xdf Linux FS Read / write Method 1 (âBesideâ or âEdgeâ) Copy to Local File Method 2: Local (Linux) parallel processing using all cores on one node, streaming data from / to HDFS Compute Context HadoopCompute Context HadoopCompute Context Local Parallel Compute Context Hadoop Linux (Local) File-System HDFS Csv, Xdf 1:n nodes 1:n disks 1:(n x number of nodes) disks 1 Edge node
- 32. Method 3 Method 3: Hadoop (Map-Reduce) parallel processing using all cores on n nodes, using HDFS data on each node Compute Context HadoopCompute Context HadoopCompute Context Local Parallel Compute Context Hadoop Linux (Local) File-System HDFS Csv, Xdf Processing Data 1:n nodes 1:n disks 1:(n x number of nodes) disks Csv, Xdf HDFS Read / write (âinsideâ) R script sent to data nodes 1 Edge node R model script sent to Master Node: 1. Starts a master process 2. Distribute work 3. Master tasks for each node 4. Master initiates distributed work 1.Hadoop schedules mapper for each split 2.Algorithm computes intermediate result 3.Reducer combines intermediate results 5. Master process evaluates completion 6. Iterates as required by the algorithm 7. Returns consolidated answer to script
- 33. DistributedR - What processing mode to use, when? Analytic data set size and processing complexity (e.g. simple summary statistics vs iterative algorithm) guide the use of Method 1 and 2 (Edge Node / Server Linux local processing) vs Method 3 (in-Hadoop processing) Low Medium High Small Data < 10GB Medium Data < 50GB Bigger Data > 50GB Edge Node Linux processing In-Hadoop processing Local Linux file-system Hadoop file-system Legend Processing Complexity Data Size
- 34. While Open Source R delivers: âĒ Capability âĒ 6500+ Algorithm & Connector Packages Available for Free in CRAN âĒ Simplicity âĒ R Skills Transfer / Lower cost of Talent âĒ Ease of Integration with Other Analytics Packages & Data âĒ Access to Huge Libraries of R Analytical Algorithms âĒ Speed âĒ Intel-Optimized Computation âĒ Peace of mind âĒ Knowledge that your business is using a stable platform backed with commercial support and services âĒ Platform longevity for more predictability around costs âĒ Speed and scalability âĒ Faster decisions using advanced analytics that were previously unachievable âĒ In-Hadoop & In Teradata Analysis âĒ Efficiency âĒ Continue getting returns on existing hardware and software investments âĒ Developers can write code once and deploy it anywhere, keeping costs low âĒ Flexibility and agility âĒ Model data in a hybrid environment: on-premises, in the cloud, or both âĒ Scripting, modeling, and in-database analytics across platforms shrinks analysis time and enables agile response to business needs SQL Server R Services and Microsoft R Server deliver:
- 36. Introducing Microsoft R Open âĒ Enhanced Open Source R distribution âĒ Based on the latestOpenSourceR (3.1.2) âĒ Built,testedanddistributed by Microsoft âĒ EnhancedbyIntelMKL Libraryto speedup linearalgebra functions âĒ Compatible with all R-related software âĒ CRANpackages,RStudio, third-partyR integrations,âĶ âĒ Revolutions Open-Source R packages âĒ ReproducibleR Toolkitâ Checkpoint, miniCRAN âĒ ParallelRâ parallelise execution viaâforeachâloop âĒ Rhadoopâ rhdfs, rhbase,ravro,rmr2, plyrmr âĒ AzureMLâ read/writedatatoAzureML,publishR code asML API âĒ MRAN website mran.revolutionanalytics.com âĒ Enhanceddocumentation andlearningresources âĒ Discover6500 free add-on Rpackages âĒ Open source (GPLv2 license) - 100% free to download, use and share
- 37. Datasize In-memory In-memory In-Memoryor Disk Based Speed of Analysis Single threaded Multi-threaded Multi-threaded, parallel processing 1:N servers Support Community Community Community + Commercial Analytic Breadth & Depth 7500+ innovative analytic packages 7500+ innovative analytic packages 7500+ innovative packages + commercial parallel high- speed functions Licence Open Source Open Source Commercial license. Supported release with indemnity CRAN, MRO, MRS Comparison Microsoft R Open Microsoft R Server
- 38. More efficient and multi-threaded math computation. Benefits math intensive processing. No benefit to program logic and data transform CRAN R compared to Microsoft R Open âĒ Matrix calculation â upto 27x faster âĒ Matrix functions â upto 16x faster âĒ Programation â 0x faster