Building Custom Machine Learning Algorithms With Apache SystemML
6 likes1,649 views
This document discusses Apache SystemML, which is a machine learning framework for building custom machine learning algorithms on Apache Spark. It originated from research projects at IBM involving machine learning on Hadoop. SystemML aims to allow data scientists to build ML solutions using languages like R and Python, while executing algorithms on big data platforms like Spark. It provides a high-level language for expressing algorithms and performs automatic parallelization and optimization. The document demonstrates SystemML through a matrix factorization example for a targeted advertising problem. It shows how to wrangle data, build a custom algorithm, and get results. In conclusion, it recommends that readers try out SystemML through its website.
1 of 30
Downloaded 113 times
Ad
Recommended
SparkML: Easy ML Productization for Real-Time Bidding
SparkML: Easy ML Productization for Real-Time BiddingDatabricks The document discusses the implementation of Spark ML at DataXu for real-time bidding in programmatic marketing, detailing challenges faced and solutions developed for efficient machine learning operations. Key issues included model readiness for low-latency environments, data partitioning, and training costs, with solutions that yielded significant performance improvements and cost savings. The transition to Spark ML was successfully achieved through careful planning and strategic testing, emphasizing the importance of adaptability and system transparency.
Machine Learning with Apache Spark
Machine Learning with Apache SparkIBM Cloud Data Services The document discusses the significance of machine learning and big data analytics using Apache Spark, highlighting its capabilities for handling massive data volumes and enhancing decision-making processes for businesses. It outlines various analytics types, machine learning algorithms available in Spark, and a demonstration concerning flight delay predictions using weather data. Additionally, it emphasizes IBM's commitment to Spark and its resources for developers in the cloud ecosystem.
Is This Thing On? A Well State Model for the People
Is This Thing On? A Well State Model for the PeopleDatabricks The document discusses using machine learning models to determine well production state (on vs off) from sensor data. It presents an existing data architecture and issues with data quality. A supervised learning model is proposed using a decision tree trained on labeled rod pump production data. The modeling workflow includes data preprocessing, feature engineering, hyperparameter tuning and grid search. Decision trees are chosen for their interpretability but the document notes larger models may perform better. Overall production state modeling could help optimize operations and outperform existing controllers.
Saving Energy in Homes with a Unified Approach to Data and AI
Saving Energy in Homes with a Unified Approach to Data and AIDatabricks Quby aims to reduce energy waste in European households by utilizing a data-driven platform that provides personalized advice to users on efficient appliance use. The company has developed a unified data analytics setup and artificial intelligence (AI) capabilities, resulting in significant energy savings, including the identification of 87 million inefficient appliance cycles and the conservation of 67 million kWh. Their approach includes continuous model training and deployment, enhancing collaboration and speeding up development cycles to support the transition to sustainable energy systems.
Breaking Down Analytical and Computational Barriers Across the Energy Industr...
Breaking Down Analytical and Computational Barriers Across the Energy Industr...Spark Summit DNV GL is presenting on breaking down barriers in energy data analytics through their work in statistical computing and machine learning. They discuss using big data from smart grids and advanced metering infrastructure to better understand demand behavior and integrate disruptive technologies like electric vehicles and solar power. DNV GL also demonstrates a pilot project using Databricks and Spark to perform real-time energy forecasting and grid reliability analysis at scale.
Lessons Learned Replatforming A Large Machine Learning Application To Apache ...
Lessons Learned Replatforming A Large Machine Learning Application To Apache ...Databricks Patrick Caldon and Taylor Hess from Morningstar Inc. presented lessons learned from replatforming a large machine learning application, emphasizing model scalability, end-to-end system testing, and making iteration easy for analysts. They outlined the development of a financial risk factor model, which analyzes securities to forecast returns, and described improvements in processing capabilities from their new architecture. The presentation highlighted the importance of local runs and efficient workflows to enhance productivity in model development and deployment.
Machine Learning Pipelines
Machine Learning Pipelinesjeykottalam This document discusses machine learning pipelines and introduces Evan Sparks' presentation on building image classification pipelines. It provides an overview of feature extraction techniques used in computer vision like normalization, patch extraction, convolution, rectification and pooling. These techniques are used to transform images into feature vectors that can be input to linear classifiers. The document encourages building simple, intermediate and advanced image classification pipelines using these techniques to qualitatively and quantitatively compare their effectiveness.
Anomaly Detection at Scale!
Anomaly Detection at Scale!Databricks The document discusses the challenges of detecting data anomalies at scale within Nielsen's marketing cloud, detailing their system architecture and cost-effective serverless solutions. Key points include the importance of low-cost storage and processing, the use of time series modeling for predictions, and the necessity of building trust in model outcomes. The session emphasizes three main themes: reducing costs, designing for scalability, and the significance of reliable prediction models.
Production ready big ml workflows from zero to hero daniel marcous @ waze
Production ready big ml workflows from zero to hero daniel marcous @ wazeIdo Shilon This document provides an overview of production-ready machine learning workflows. It discusses challenges of big ML including skill gaps, dimensionality, and model complexity. The solution is presented as a workflow that includes preprocessing, naive implementation, monitoring with dashboards, optimization, A/B testing, and iteration. Key steps are to measure first before optimizing, start small and grow, test infrastructure, and establish a baseline before optimizing models. The document provides examples of applying these workflows at Waze for tasks like irregular traffic event detection, dangerous place identification, and speed limit inference.
Multi Model Machine Learning by Maximo Gurmendez and Beth Logan
Multi Model Machine Learning by Maximo Gurmendez and Beth LoganSpark Summit 1. DataXu uses Spark for large-scale machine learning to power real-time bidding for display ads, processing 2 petabytes of data daily and making 1.6 million bid decisions per second across 5 continents.
2. Some challenges in using Spark included smart dataset partitioning, handling categorical and functional features, and enabling real-time model instantiations for bidding.
3. Spark SQL and UDFs helped address these challenges by enabling declarative feature encoding and selection, reusable transformations for both training and bidding, and efficient categorical encoding and top K feature value extraction.
Building Identity Graphs over Heterogeneous Data
Building Identity Graphs over Heterogeneous DataDatabricks The document discusses the construction and scaling of identity graphs using heterogeneous data to achieve coherent customer identities across multiple channels. It outlines the benefits of using graph structures over traditional table joins, detailing the stages of building and processing identity linkages and the challenges faced, such as data quality issues and the need for real-time updates. Additionally, the paper emphasizes the importance of handling vast amounts of data efficiently while maintaining both precision and coverage in identity resolution.
AI Modernization at AT&T and the Application to Fraud with Databricks
AI Modernization at AT&T and the Application to Fraud with DatabricksDatabricks The document discusses AT&T's approach to AI modernization, focusing on its application in combating fraud using Databricks. It outlines the evolution of AI at AT&T, showcases various fraud types affecting the company, and details the deployment of machine learning models for real-time fraud detection. Additionally, it highlights the development of a feature store and governance mechanisms to enhance AI model performance and oversight.
Auto-Train a Time-Series Forecast Model With AML + ADB
Auto-Train a Time-Series Forecast Model With AML + ADBDatabricks The document outlines a presentation by Moe Steller on auto-training time-series forecast models using Azure Machine Learning (AML) and Azure Databricks (ADB). It covers the capabilities of various forecasting algorithms, product previews, advanced parameters, and the architecture that supports their integration. Additionally, it emphasizes automation in model development and operationalization for various industries including healthcare, finance, and retail.
Data-Driven Transformation: Leveraging Big Data at Showtime with Apache Spark
Data-Driven Transformation: Leveraging Big Data at Showtime with Apache SparkDatabricks Showtime Networks Inc. leverages big data through its standalone streaming service to understand subscriber behaviors and optimize content offerings, employing data science techniques and machine learning models. A small team of data scientists enhances data accessibility and quality by using tools like Apache Spark, Airflow, and Delta for efficient data management. Their efforts aim to improve subscriber retention, reduce churn, and ultimately drive revenue growth through data-driven decision-making.
Distributed Time Travel for Feature Generation by DB Tsai and Prasanna Padman...
Distributed Time Travel for Feature Generation by DB Tsai and Prasanna Padman...Spark Summit This document describes Netflix's use of distributed time travel for feature generation using data snapshots. Key points:
1. Netflix uses data snapshots of online services stored in S3 to generate features offline for model training and experimentation, allowing ideas to be tested on historical data quickly before deploying live tests.
2. A "DeLorean" system selects contexts, takes snapshots of data from services like viewing history and playlists, and provides batch APIs to access snapshot data for offline experiments.
3. Feature encoders generate features using the snapshot data without calling live systems, and features are stored in Parquet files in S3. Successful models are then deployed online.
4. This approach significantly reduces the time
Extending Machine Learning Algorithms with PySpark
Extending Machine Learning Algorithms with PySparkDatabricks 1. The document discusses using PySpark and Pandas UDFs to perform machine learning at scale for genomic data. It describes a genomics use case called GloWGR that uses this approach.
2. Three key problems are identified with existing tools: genomic data is growing too quickly; bioinformaticians are unfamiliar with Scala; and ML algorithms are difficult to write in Spark SQL. The solutions proposed are to use Spark, provide a Python client, and write algorithms in Python linked to Spark.
3. GloWGR is presented as a novel whole genome regression and association study algorithm built with PySpark. It uses Pandas UDFs to parallelize the REGENIE method and perform tasks like dimensionality
Puree through Trillion of clicks in seconds using Interana
Puree through Trillion of clicks in seconds using InteranaJagjit Srawan The document outlines Interana's capabilities in handling big data, focusing on event stream behavior analytics with a full-stack solution for real-time insights. It discusses the technical architecture, including data ingestion, performance optimization with columnar storage, and operational approach in managed environments like AWS and Azure. Additionally, it highlights various customer use cases and the importance of sampling and data redundancy in analysis.
Experimental Design for Distributed Machine Learning with Myles Baker
Experimental Design for Distributed Machine Learning with Myles BakerDatabricks This document discusses experimental design for distributed machine learning models. It outlines common problems in machine learning modeling like selecting the best algorithm and evaluating a model's expected generalization error. It describes steps in a machine learning study like collecting data, building models, and designing experiments. The goal of experimentation is to understand how model factors affect outcomes and obtain statistically significant conclusions. Techniques discussed for analyzing distributed model outputs include precision-recall curves, confusion matrices, and hypothesis testing methods like the chi-squared test and McNemar's test. The document emphasizes that experimental design for distributed learning poses new challenges around data characteristics, computational complexity, and reproducing results across models.
Simplify Distributed TensorFlow Training for Fast Image Categorization at Sta...
Simplify Distributed TensorFlow Training for Fast Image Categorization at Sta...Databricks This document discusses the use of Horovodrunner at Starbucks to simplify deep learning processes, particularly in distributed environments. The presenters, Denny Lee and Vishwanath Subramanian, emphasize the importance of providing efficient analytics solutions and facilitating machine learning capabilities for large-scale data processing. Key features include on-demand infrastructure provisioning, secured data connectivity, and templates for deploying experiments and applications.
Scalable Deep Learning Platform On Spark In Baidu
Scalable Deep Learning Platform On Spark In BaiduJen Aman This document summarizes Baidu's work on scaling deep learning using Spark. It discusses:
1) Baidu's goals of implementing Spark ML abstractions to train deep learning models with minimal code changes and leveraging Paddle's distributed training capabilities.
2) The system architecture which uses Spark and YARN for resource management and Paddle for distributed training.
3) Experiments showing Paddle can efficiently train large models like image recognition on ImageNet using many machines and GPUs, as well as sparse models.
Machine learning model to production
Machine learning model to productionGeorg Heiler This document discusses moving machine learning models from prototype to production. It outlines some common problems with the current workflow where moving to production often requires redevelopment from scratch. Some proposed solutions include using notebooks as APIs and developing analytics that are accessed via an API. It also discusses different data science platforms and architectures for building end-to-end machine learning systems, focusing on flexibility, security, testing and scalability for production environments. The document recommends a custom backend integrated with Spark via APIs as the best approach for the current project.
Using GraphX/Pregel on Browsing History to Discover Purchase Intent by Lisa Z...
Using GraphX/Pregel on Browsing History to Discover Purchase Intent by Lisa Z...Spark Summit This document discusses using graph-based machine learning on browsing history data to discover customer purchase intent for advertisers. It presents challenges with existing solutions like SVD that identify general online buyers but not advertiser-specific patterns. The document proposes representing sites as a graph and using GraphX's Pregel API to propagate positive customer labels along site connections, assigning higher scores to similar sites. Evaluation shows this approach identifies advertiser-relevant sites while addressing issues like model sparsity and frequency. It also provides lessons learned on optimizing Spark jobs.
Apache Spark for Machine Learning with High Dimensional Labels: Spark Summit ...
Apache Spark for Machine Learning with High Dimensional Labels: Spark Summit ...Spark Summit The document discusses the use of Apache Spark for machine learning in the context of high dimensional labels related to television advertising at Cadent. It outlines a unique data infrastructure that leverages analytical algorithms and dimensionality reduction techniques to optimize decision-making based on viewer ratings and other variables. The authors, Michael Zargham and Stefan Panayotov, highlight their innovative approach to predictive modeling using gradient boosted tree regressors and principal component analysis.
How Spark Enables the Internet of Things- Paula Ta-Shma
How Spark Enables the Internet of Things- Paula Ta-ShmaSpark Summit The document summarizes an IBM research paper on how Spark can enable Internet of Things (IoT) use cases. It describes an IoT architecture used for a smart city use case with Madrid city buses. Data is collected from 3000 traffic sensors into Kafka and aggregated into Swift objects using Secor. Spark is used to access and analyze the data to detect traffic patterns and inform bus routing decisions in real-time. The system aims to improve customer satisfaction and reduce costs by responding efficiently to traffic issues.
Ray: Enterprise-Grade, Distributed Python
Ray: Enterprise-Grade, Distributed PythonDatabricks This document discusses Ray, a distributed Python framework designed to handle complex computing requirements driven by machine learning and AI workloads. It covers Ray's advantages over Spark in handling non-uniform data and tasks, and provides guidance on how to get started using Ray alongside existing Python libraries. Additionally, it emphasizes the importance of user feedback and offers resources for community engagement and learning about Ray.
Build Real-Time Applications with Databricks Streaming
Build Real-Time Applications with Databricks StreamingDatabricks This document discusses using Databricks, Spark, and Power BI for real-time data streaming. It describes a use case of a fire department needing real-time reporting of equipment locations, personnel statuses, and active incidents. The solution involves ingesting event data using Azure Event Hubs, processing the stream using Databricks and Spark Structured Streaming, storing the results in Delta Lake, and visualizing the data in Power BI dashboards. It then demonstrates the architecture by walking through creating Delta tables, streaming from Event Hubs to Delta Lake, and running a sample event simulator.
No REST till Production – Building and Deploying 9 Models to Production in 3 ...
No REST till Production – Building and Deploying 9 Models to Production in 3 ...Databricks Charmee Patel from Syntasa discusses building and deploying 9 models to production in 3 weeks to support media buying decisions for certain product segments using clickstream and enterprise data from ~2M visitors and ~100K SKUs. Key challenges included high data volume, complexity, non-stationarity, and reliability in production. Syntasa addressed this through an experiment process template, feature store, and ensemble modeling approach. Results showed significant lift over rule-based approaches, with the bespoke algorithmic models driving much higher conversion rates and marketing activity.
Application and Challenges of Streaming Analytics and Machine Learning on Mu...
Application and Challenges of Streaming Analytics and Machine Learning on Mu...Databricks The document outlines the application and challenges of streaming analytics and machine learning in smart manufacturing, specifically in the pharmaceutical sector, highlighting the work of Quartic.ai in industrial IoT. Key achievements include improving reliability and uptime of critical equipment through predictive health monitoring and smart automation solutions. The document also discusses technical challenges such as late data arrival and model deployment on heterogeneous data streams.
Spark And Cassandra: 2 Fast, 2 Furious
Spark And Cassandra: 2 Fast, 2 FuriousJen Aman The document discusses the Datastax Spark Cassandra Connector. It provides an overview of how the connector allows Spark to interact with Cassandra data, including performing full table scans, pushing down filters and projections to Cassandra, distributed joins using Cassandra's partitioning, and writing data back to Cassandra in a distributed way. It also highlights some recent features of the connector like support for Cassandra 3.0, materialized views, and performance improvements from the Java Wildcard Cassandra Tester project.
Regression using Apache SystemML by Alexandre V Evfimievski
Regression using Apache SystemML by Alexandre V EvfimievskiArvind Surve The document provides an overview of linear regression and its components, such as least squares, matrix notation, and computation methods for large datasets. It discusses the conjugate gradient method as a solution for large matrices, the importance of regularization to avoid issues with linearly dependent features, and provides code snippets and statistical views to improve understanding and performance of regression models. Concepts like the maximum likelihood estimator, variance, and r-squared statistics are also highlighted, emphasizing the computational and theoretical aspects of linear regression.
More Related Content
What's hot (20)
Production ready big ml workflows from zero to hero daniel marcous @ waze
Production ready big ml workflows from zero to hero daniel marcous @ wazeIdo Shilon This document provides an overview of production-ready machine learning workflows. It discusses challenges of big ML including skill gaps, dimensionality, and model complexity. The solution is presented as a workflow that includes preprocessing, naive implementation, monitoring with dashboards, optimization, A/B testing, and iteration. Key steps are to measure first before optimizing, start small and grow, test infrastructure, and establish a baseline before optimizing models. The document provides examples of applying these workflows at Waze for tasks like irregular traffic event detection, dangerous place identification, and speed limit inference.
Multi Model Machine Learning by Maximo Gurmendez and Beth Logan
Multi Model Machine Learning by Maximo Gurmendez and Beth LoganSpark Summit 1. DataXu uses Spark for large-scale machine learning to power real-time bidding for display ads, processing 2 petabytes of data daily and making 1.6 million bid decisions per second across 5 continents.
2. Some challenges in using Spark included smart dataset partitioning, handling categorical and functional features, and enabling real-time model instantiations for bidding.
3. Spark SQL and UDFs helped address these challenges by enabling declarative feature encoding and selection, reusable transformations for both training and bidding, and efficient categorical encoding and top K feature value extraction.
Building Identity Graphs over Heterogeneous Data
Building Identity Graphs over Heterogeneous DataDatabricks The document discusses the construction and scaling of identity graphs using heterogeneous data to achieve coherent customer identities across multiple channels. It outlines the benefits of using graph structures over traditional table joins, detailing the stages of building and processing identity linkages and the challenges faced, such as data quality issues and the need for real-time updates. Additionally, the paper emphasizes the importance of handling vast amounts of data efficiently while maintaining both precision and coverage in identity resolution.
AI Modernization at AT&T and the Application to Fraud with Databricks
AI Modernization at AT&T and the Application to Fraud with DatabricksDatabricks The document discusses AT&T's approach to AI modernization, focusing on its application in combating fraud using Databricks. It outlines the evolution of AI at AT&T, showcases various fraud types affecting the company, and details the deployment of machine learning models for real-time fraud detection. Additionally, it highlights the development of a feature store and governance mechanisms to enhance AI model performance and oversight.
Auto-Train a Time-Series Forecast Model With AML + ADB
Auto-Train a Time-Series Forecast Model With AML + ADBDatabricks The document outlines a presentation by Moe Steller on auto-training time-series forecast models using Azure Machine Learning (AML) and Azure Databricks (ADB). It covers the capabilities of various forecasting algorithms, product previews, advanced parameters, and the architecture that supports their integration. Additionally, it emphasizes automation in model development and operationalization for various industries including healthcare, finance, and retail.
Data-Driven Transformation: Leveraging Big Data at Showtime with Apache Spark
Data-Driven Transformation: Leveraging Big Data at Showtime with Apache SparkDatabricks Showtime Networks Inc. leverages big data through its standalone streaming service to understand subscriber behaviors and optimize content offerings, employing data science techniques and machine learning models. A small team of data scientists enhances data accessibility and quality by using tools like Apache Spark, Airflow, and Delta for efficient data management. Their efforts aim to improve subscriber retention, reduce churn, and ultimately drive revenue growth through data-driven decision-making.
Distributed Time Travel for Feature Generation by DB Tsai and Prasanna Padman...
Distributed Time Travel for Feature Generation by DB Tsai and Prasanna Padman...Spark Summit This document describes Netflix's use of distributed time travel for feature generation using data snapshots. Key points:
1. Netflix uses data snapshots of online services stored in S3 to generate features offline for model training and experimentation, allowing ideas to be tested on historical data quickly before deploying live tests.
2. A "DeLorean" system selects contexts, takes snapshots of data from services like viewing history and playlists, and provides batch APIs to access snapshot data for offline experiments.
3. Feature encoders generate features using the snapshot data without calling live systems, and features are stored in Parquet files in S3. Successful models are then deployed online.
4. This approach significantly reduces the time
Extending Machine Learning Algorithms with PySpark
Extending Machine Learning Algorithms with PySparkDatabricks 1. The document discusses using PySpark and Pandas UDFs to perform machine learning at scale for genomic data. It describes a genomics use case called GloWGR that uses this approach.
2. Three key problems are identified with existing tools: genomic data is growing too quickly; bioinformaticians are unfamiliar with Scala; and ML algorithms are difficult to write in Spark SQL. The solutions proposed are to use Spark, provide a Python client, and write algorithms in Python linked to Spark.
3. GloWGR is presented as a novel whole genome regression and association study algorithm built with PySpark. It uses Pandas UDFs to parallelize the REGENIE method and perform tasks like dimensionality
Puree through Trillion of clicks in seconds using Interana
Puree through Trillion of clicks in seconds using InteranaJagjit Srawan The document outlines Interana's capabilities in handling big data, focusing on event stream behavior analytics with a full-stack solution for real-time insights. It discusses the technical architecture, including data ingestion, performance optimization with columnar storage, and operational approach in managed environments like AWS and Azure. Additionally, it highlights various customer use cases and the importance of sampling and data redundancy in analysis.
Experimental Design for Distributed Machine Learning with Myles Baker
Experimental Design for Distributed Machine Learning with Myles BakerDatabricks This document discusses experimental design for distributed machine learning models. It outlines common problems in machine learning modeling like selecting the best algorithm and evaluating a model's expected generalization error. It describes steps in a machine learning study like collecting data, building models, and designing experiments. The goal of experimentation is to understand how model factors affect outcomes and obtain statistically significant conclusions. Techniques discussed for analyzing distributed model outputs include precision-recall curves, confusion matrices, and hypothesis testing methods like the chi-squared test and McNemar's test. The document emphasizes that experimental design for distributed learning poses new challenges around data characteristics, computational complexity, and reproducing results across models.
Simplify Distributed TensorFlow Training for Fast Image Categorization at Sta...
Simplify Distributed TensorFlow Training for Fast Image Categorization at Sta...Databricks This document discusses the use of Horovodrunner at Starbucks to simplify deep learning processes, particularly in distributed environments. The presenters, Denny Lee and Vishwanath Subramanian, emphasize the importance of providing efficient analytics solutions and facilitating machine learning capabilities for large-scale data processing. Key features include on-demand infrastructure provisioning, secured data connectivity, and templates for deploying experiments and applications.
Scalable Deep Learning Platform On Spark In Baidu
Scalable Deep Learning Platform On Spark In BaiduJen Aman This document summarizes Baidu's work on scaling deep learning using Spark. It discusses:
1) Baidu's goals of implementing Spark ML abstractions to train deep learning models with minimal code changes and leveraging Paddle's distributed training capabilities.
2) The system architecture which uses Spark and YARN for resource management and Paddle for distributed training.
3) Experiments showing Paddle can efficiently train large models like image recognition on ImageNet using many machines and GPUs, as well as sparse models.
Machine learning model to production
Machine learning model to productionGeorg Heiler This document discusses moving machine learning models from prototype to production. It outlines some common problems with the current workflow where moving to production often requires redevelopment from scratch. Some proposed solutions include using notebooks as APIs and developing analytics that are accessed via an API. It also discusses different data science platforms and architectures for building end-to-end machine learning systems, focusing on flexibility, security, testing and scalability for production environments. The document recommends a custom backend integrated with Spark via APIs as the best approach for the current project.
Using GraphX/Pregel on Browsing History to Discover Purchase Intent by Lisa Z...
Using GraphX/Pregel on Browsing History to Discover Purchase Intent by Lisa Z...Spark Summit This document discusses using graph-based machine learning on browsing history data to discover customer purchase intent for advertisers. It presents challenges with existing solutions like SVD that identify general online buyers but not advertiser-specific patterns. The document proposes representing sites as a graph and using GraphX's Pregel API to propagate positive customer labels along site connections, assigning higher scores to similar sites. Evaluation shows this approach identifies advertiser-relevant sites while addressing issues like model sparsity and frequency. It also provides lessons learned on optimizing Spark jobs.
Apache Spark for Machine Learning with High Dimensional Labels: Spark Summit ...
Apache Spark for Machine Learning with High Dimensional Labels: Spark Summit ...Spark Summit The document discusses the use of Apache Spark for machine learning in the context of high dimensional labels related to television advertising at Cadent. It outlines a unique data infrastructure that leverages analytical algorithms and dimensionality reduction techniques to optimize decision-making based on viewer ratings and other variables. The authors, Michael Zargham and Stefan Panayotov, highlight their innovative approach to predictive modeling using gradient boosted tree regressors and principal component analysis.
How Spark Enables the Internet of Things- Paula Ta-Shma
How Spark Enables the Internet of Things- Paula Ta-ShmaSpark Summit The document summarizes an IBM research paper on how Spark can enable Internet of Things (IoT) use cases. It describes an IoT architecture used for a smart city use case with Madrid city buses. Data is collected from 3000 traffic sensors into Kafka and aggregated into Swift objects using Secor. Spark is used to access and analyze the data to detect traffic patterns and inform bus routing decisions in real-time. The system aims to improve customer satisfaction and reduce costs by responding efficiently to traffic issues.
Ray: Enterprise-Grade, Distributed Python
Ray: Enterprise-Grade, Distributed PythonDatabricks This document discusses Ray, a distributed Python framework designed to handle complex computing requirements driven by machine learning and AI workloads. It covers Ray's advantages over Spark in handling non-uniform data and tasks, and provides guidance on how to get started using Ray alongside existing Python libraries. Additionally, it emphasizes the importance of user feedback and offers resources for community engagement and learning about Ray.
Build Real-Time Applications with Databricks Streaming
Build Real-Time Applications with Databricks StreamingDatabricks This document discusses using Databricks, Spark, and Power BI for real-time data streaming. It describes a use case of a fire department needing real-time reporting of equipment locations, personnel statuses, and active incidents. The solution involves ingesting event data using Azure Event Hubs, processing the stream using Databricks and Spark Structured Streaming, storing the results in Delta Lake, and visualizing the data in Power BI dashboards. It then demonstrates the architecture by walking through creating Delta tables, streaming from Event Hubs to Delta Lake, and running a sample event simulator.
No REST till Production – Building and Deploying 9 Models to Production in 3 ...
No REST till Production – Building and Deploying 9 Models to Production in 3 ...Databricks Charmee Patel from Syntasa discusses building and deploying 9 models to production in 3 weeks to support media buying decisions for certain product segments using clickstream and enterprise data from ~2M visitors and ~100K SKUs. Key challenges included high data volume, complexity, non-stationarity, and reliability in production. Syntasa addressed this through an experiment process template, feature store, and ensemble modeling approach. Results showed significant lift over rule-based approaches, with the bespoke algorithmic models driving much higher conversion rates and marketing activity.
Application and Challenges of Streaming Analytics and Machine Learning on Mu...
Application and Challenges of Streaming Analytics and Machine Learning on Mu...Databricks The document outlines the application and challenges of streaming analytics and machine learning in smart manufacturing, specifically in the pharmaceutical sector, highlighting the work of Quartic.ai in industrial IoT. Key achievements include improving reliability and uptime of critical equipment through predictive health monitoring and smart automation solutions. The document also discusses technical challenges such as late data arrival and model deployment on heterogeneous data streams.
Viewers also liked (20)
Spark And Cassandra: 2 Fast, 2 Furious
Spark And Cassandra: 2 Fast, 2 FuriousJen Aman The document discusses the Datastax Spark Cassandra Connector. It provides an overview of how the connector allows Spark to interact with Cassandra data, including performing full table scans, pushing down filters and projections to Cassandra, distributed joins using Cassandra's partitioning, and writing data back to Cassandra in a distributed way. It also highlights some recent features of the connector like support for Cassandra 3.0, materialized views, and performance improvements from the Java Wildcard Cassandra Tester project.
Regression using Apache SystemML by Alexandre V Evfimievski
Regression using Apache SystemML by Alexandre V EvfimievskiArvind Surve The document provides an overview of linear regression and its components, such as least squares, matrix notation, and computation methods for large datasets. It discusses the conjugate gradient method as a solution for large matrices, the importance of regularization to avoid issues with linearly dependent features, and provides code snippets and statistical views to improve understanding and performance of regression models. Concepts like the maximum likelihood estimator, variance, and r-squared statistics are also highlighted, emphasizing the computational and theoretical aspects of linear regression.
Inside Apache SystemML by Frederick Reiss
Inside Apache SystemML by Frederick ReissSpark Summit 1. The document describes the origins and goals of the SystemML project for scalable machine learning.
2. SystemML was created to allow data scientists to write machine learning algorithms in R and automatically compile and optimize them to run efficiently on large datasets in parallel.
3. An example alternating least squares algorithm is shown written concisely in R, while traditional approaches required translating algorithms to other languages like Scala which was error-prone and slowed iteration. SystemML aims to allow the same algorithm to run fast at large scale with the same answer.
Spark Summit EU talk by Heiko Korndorf
Spark Summit EU talk by Heiko KorndorfSpark Summit Heiko Korndorf presented on scaling SparkR in production and lessons from real-world projects. The presentation covered classifying SparkR as both a data science and data engineering tool. It discussed SparkR's architecture in versions 1.x and 2.x, approaches to parallelizing R code with Spark 1.5/1.6 and YARN, and dynamic R deployment including dependencies. Advanced techniques like 2-level parallelization using GPGPU were also presented. The talk concluded with an outlook on further integrating data engineering and data science and new technical approaches like simplifying data pipelines and moving calculations to GPUs.
Yggdrasil: Faster Decision Trees Using Column Partitioning In Spark
Yggdrasil: Faster Decision Trees Using Column Partitioning In SparkJen Aman This document introduces Yggdrasil, a new approach for training decision trees in Spark that partitions data by column instead of by row. Column partitioning significantly reduces communication costs for deep trees with many features. Evaluation on real-world datasets with millions of rows and thousands of features shows Yggdrasil achieves up to 24x speedup over the existing row-partitioning approach in Spark MLlib. The authors propose merging Yggdrasil into Spark MLlib to provide both row and column partitioning options for optimal performance on different problem sizes and depths.
RISELab:Enabling Intelligent Real-Time Decisions
RISELab:Enabling Intelligent Real-Time DecisionsJen Aman The RISE Lab at Berkeley aims to develop an open-source data analytics stack for real-time intelligent decision-making using live data, with a focus on speed, security, and personalization. The initiative involves research in algorithms, machine learning, and systems to create frameworks that improve the responsiveness and robustness of decision systems. The project's ongoing efforts include enhancing existing tools like Apache Spark and exploring secure reinforcement learning applications.
Huawei Advanced Data Science With Spark Streaming
Huawei Advanced Data Science With Spark StreamingJen Aman This document discusses streamDM, an open source machine learning library for stream mining in Spark Streaming. It summarizes streamDM's capabilities for incremental learning on data streams using algorithms like SGD, Naive Bayes, clustering and decision trees. Examples of using streamDM in Huawei's network alarm analysis and fault localization systems are provided, demonstrating improvements in efficiency, accuracy and ability to handle large volumes of streaming data. The document encourages researchers to apply for Huawei's Innovation Research Program grants to further collaborative work on stream mining algorithms and applications.
Low Latency Execution For Apache Spark
Low Latency Execution For Apache SparkJen Aman This document describes Drizzle, a low latency execution engine for Apache Spark. It addresses the high overheads of Spark's centralized scheduling model by decoupling execution from scheduling through batch scheduling and pre-scheduling of shuffles. Microbenchmarks show Drizzle achieves milliseconds latency for iterative workloads compared to hundreds of milliseconds for Spark. End-to-end experiments show Drizzle improves latency for streaming and machine learning workloads like logistic regression. The authors are working on automatic batch tuning and an open source release of Drizzle.
Re-Architecting Spark For Performance Understandability
Re-Architecting Spark For Performance UnderstandabilityJen Aman The document discusses a new architecture called "monotasks" that is designed to make it easier to reason about the performance of Apache Spark jobs. The key aspects are:
1) Each task in a Spark job is dedicated to using a single resource (e.g. network, CPU, disk).
2) Dedicated schedulers control contention between tasks for resources.
3) The timing of individual "monotasks" can be used to model an ideal performance and understand bottlenecks.
Livy: A REST Web Service For Apache Spark
Livy: A REST Web Service For Apache SparkJen Aman Livy is an open source REST service for interacting with and managing Spark contexts and jobs. It allows clients to submit Spark jobs via REST, monitor their status, and retrieve results. Livy manages long-running Spark contexts in a cluster and supports running multiple independent contexts simultaneously from different clients. It provides client APIs in Java, Scala, and soon Python to interface with the Livy REST endpoints for submitting, monitoring, and retrieving results of Spark jobs.
Spark Uber Development Kit
Spark Uber Development KitJen Aman This document summarizes a presentation given at Spark Summit 2016 about tools and techniques used at Uber for Spark development and jobs. It introduces SCBuilder for encapsulating cluster environments, Kafka dispersal for publishing RDD results to Kafka, and SparkPlug for kickstarting job development with templates. It also discusses SparkChamber for distributed log debugging and future work including analytics, machine learning, and resource usage auditing.
Spark on Mesos
Spark on MesosJen Aman This document summarizes a talk on Spark on Mesos given by Dean Wampler from Lightbend and Timothy Chen from Mirantis. It discusses why Spark on Mesos is useful by allowing one cluster system to run multiple tools. It then covers recent updates like a new integration test suite, coarse-grained scheduler improvements, and Mesos framework authentication. Finally, it outlines future plans such as GPU support on Mesos and making "production" use of Spark on Mesos easier.
Recent Developments In SparkR For Advanced Analytics
Recent Developments In SparkR For Advanced AnalyticsDatabricks The document discusses the SparkR package, which bridges the gap between R and big data analytics, providing descriptive and predictive analytics on large datasets. It outlines the architecture, features, and implementation of SparkR, including data frame operations, statistical computations, and integration with Spark's machine learning library. Future directions for SparkR include improving API consistency, enhancing R formula support, and integrating more machine learning algorithms.
Scaling Machine Learning To Billions Of Parameters
Scaling Machine Learning To Billions Of ParametersJen Aman This document summarizes scaling machine learning to billions of parameters using Spark and a parameter server architecture. It describes the requirements for supporting both batch and sequential optimization at web scale. It then outlines the Spark + Parameter server approach, leveraging Spark for distributed processing and the parameter server for synchronizing model updates. Examples of distributed L-BFGS and Word2Vec training are presented to illustrate batch and sequential optimization respectively.
Inside Apache SystemML
Inside Apache SystemMLFrederick Reiss The document discusses the evolution and capabilities of Apache SystemML under the direction of Fred Reiss at IBM, particularly focusing on its implementation of machine learning algorithms such as Alternating Least Squares (ALS). It highlights the advantages of SystemML in terms of performance efficiency compared to other machine learning frameworks and the innovative optimizations it employs to enhance execution speed and resource utilization. The performance comparisons demonstrate SystemML's ability to execute algorithms quickly and effectively, utilizing parallel processing without the need for extensive performance code.
Airstream: Spark Streaming At Airbnb
Airstream: Spark Streaming At AirbnbJen Aman This document discusses Airbnb's data infrastructure and use of AirStream. It describes how AirStream provides a unified platform for both streaming and batch data processing using Spark SQL and a shared state store in HBase. Case studies show how AirStream is used for real-time data ingestion from Kafka to HBase, streaming exports from databases to HBase, and point-in-time queries. The document also covers how AirStream scales jobs using YARN, provides fault tolerance through checkpointing and job restarts, and monitors jobs with AirStream listeners.
Elasticsearch And Apache Lucene For Apache Spark And MLlib
Elasticsearch And Apache Lucene For Apache Spark And MLlibJen Aman This document summarizes a presentation about using Elasticsearch and Lucene for text processing and machine learning pipelines in Apache Spark. Some key points:
- Elasticsearch provides text analysis capabilities through Lucene and can be used to clean, tokenize, and vectorize text for machine learning tasks.
- Elasticsearch integrates natively with Spark through Java/Scala APIs and allows indexing and querying data from Spark.
- A typical machine learning pipeline for text classification in Spark involves tokenization, feature extraction (e.g. hashing), and a classifier like logistic regression.
- The presentation proposes preparing text analysis specifications in Elasticsearch once and reusing them across multiple Spark pipelines to simplify the workflows and avoid data movement between systems
Structuring Apache Spark 2.0: SQL, DataFrames, Datasets And Streaming - by Mi...
Structuring Apache Spark 2.0: SQL, DataFrames, Datasets And Streaming - by Mi...Databricks This document summarizes key aspects of structuring computation and data in Apache Spark using SQL, DataFrames, and Datasets. It discusses how structuring computation and data through these APIs enables optimizations like predicate pushdown and efficient joins. It also describes how data is encoded efficiently in Spark's internal format and how encoders translate between domain objects and Spark's internal representations. Finally, it introduces structured streaming as a high-level streaming API built on top of Spark SQL that allows running the same queries continuously on streaming data.
Apache Spark 2.0: A Deep Dive Into Structured Streaming - by Tathagata Das
Apache Spark 2.0: A Deep Dive Into Structured Streaming - by Tathagata Das Databricks The document discusses structured streaming in Apache Spark, focusing on its development, features, and enhancements introduced in Spark 2.0 and beyond. It highlights the transition from traditional streaming apps to a unified model that simplifies the handling of batch and streaming data while offering advanced capabilities like continuous processing and fault tolerance. The presentation emphasizes practical use cases in IoT and machine learning, showcasing how structured streaming improves analytics and query management.
Overview of Apache SystemML by Berthold Reinwald and Nakul Jindal
Overview of Apache SystemML by Berthold Reinwald and Nakul JindalArvind Surve Apache SystemML is an open-source machine learning system that provides a scalar language for data scientists to implement scalable ML algorithms with R-like and Python-like syntax. It includes a variety of pre-implemented algorithms, a cost-based optimizer for execution planning, and supports execution across single and clustered environments. The system enables high productivity for data scientists and efficient handling of large datasets with a focus on performance and scalability.
Ad
Similar to Building Custom Machine Learning Algorithms With Apache SystemML (20)
SystemML - Datapalooza Denver - 05.17.16 MWD
SystemML - Datapalooza Denver - 05.17.16 MWDMike Dusenberry Apache SystemML is a machine learning framework that allows users to write machine learning algorithms in a declarative way using a high-level language similar to R or Python. It includes a compiler and optimizer that can translate the high-level code into optimized low-level operations. The framework supports both single-machine and distributed execution using Spark. Current work is focused on improving usability for deep learning tasks and adding support for GPUs and new backends like Flink.
System mldl meetup
System mldl meetupGanesan Narayanasamy This document summarizes a presentation on Apache SystemML, an open source machine learning framework that provides scalable machine learning capabilities. It discusses SystemML's support for deep learning algorithms like convolutional neural networks and its ability to optimize machine learning workloads through techniques like operator fusion. It also demonstrates SystemML running image classification and medical image segmentation deep learning models on IBM Power systems and provides performance comparisons between Power and x86 architectures.
What's new in Apache SystemML - Declarative Machine Learning
What's new in Apache SystemML - Declarative Machine LearningLuciano Resende The document discusses the IBM Spark Technology Center and its contributions to Apache Spark and SystemML, focusing on machine learning advancements and open source leadership. It outlines the history, achievements, and technical capabilities of the Spark Technology Center, highlighting projects such as SystemML and its effectiveness in optimizing machine learning algorithms. Additionally, it emphasizes the integration of various programming languages and APIs for broader applicability in data science tasks.
Apache SystemML - Declarative Large-Scale Machine Learning
Apache SystemML - Declarative Large-Scale Machine LearningRomeo Kienzler Apache SystemML is a declarative machine learning library that:
1) Allows users to write custom machine learning algorithms declaratively without needing to handle performance or distribution details.
2) Generates and optimizes low-level execution plans to transparently distribute and run algorithms at scale on data-parallel frameworks like Spark.
3) Originally started as projects at IBM Research to handle machine learning on Hadoop and address how data scientists create ML solutions, and has since become an Apache open-source project.
2018 03 25 system ml ai and openpower meetup
2018 03 25 system ml ai and openpower meetupGanesan Narayanasamy SystemML is an Apache project that provides a declarative machine learning language for data scientists. It aims to simplify the development of custom machine learning algorithms and enable scalable execution on everything from single nodes to clusters. SystemML provides pre-implemented machine learning algorithms, APIs for various languages, and a cost-based optimizer to compile execution plans tailored to workload and hardware characteristics in order to maximize performance.
System mldl meetup
System mldl meetupGanesan Narayanasamy Apache SystemML is a machine learning language and framework that aims to simplify the life of data scientists. It allows data scientists to implement custom machine learning algorithms and run them at scale either scaled up on a single node or scaled out on clusters. SystemML uses techniques like operator fusion and rewriting to optimize machine learning workloads for both memory-bound and compute-bound problems to balance data transfer and parallelism.
Alpine Tech Talk: System ML by Berthold Reinwald
Alpine Tech Talk: System ML by Berthold ReinwaldChester Chen This document describes SystemML, an open source platform for scalable machine learning. It discusses:
- SystemML's ability to express machine learning algorithms declaratively and optimize execution plans for different environments and datasets.
- Example use cases from various industries where customers have used SystemML to solve large-scale machine learning problems.
- An example Java code implementing Gaussian non-negative matrix factorization in SystemML to factorize a large matrix.
SystemML - Declarative Machine Learning
SystemML - Declarative Machine LearningLuciano Resende The document discusses Apache SystemML, a declarative machine learning language that can run scalable machine learning algorithms on Apache Spark. It describes how SystemML allows data scientists to write algorithms in a simple R-like syntax and have them automatically compiled and run efficiently on big data in Spark. The speaker then demonstrates an alternating least squares algorithm written in R and how SystemML can compile and run the same algorithm at scale on Spark with no additional code.
Overview of Apache SystemML by Berthold Reinwald and Nakul Jindal
Overview of Apache SystemML by Berthold Reinwald and Nakul JindalArvind Surve This document provides an overview of Apache SystemML, an open source machine learning framework for scalable machine learning. It discusses how SystemML allows data scientists to implement machine learning algorithms using a declarative language, and how SystemML then compiles and optimizes the algorithms to run efficiently on everything from single nodes to large clusters. It also provides examples of DML code used in SystemML and how to invoke SystemML through its APIs or command line.
Apache SystemML 2016 Summer class primer by Berthold Reinwald
Apache SystemML 2016 Summer class primer by Berthold ReinwaldArvind Surve This document provides details about an Apache SystemML class being offered in the summer of 2016. The class aims to teach scalable machine learning using Apache SystemML. It will cover SystemML usage, hands-on exercises for developing machine learning algorithms, and advanced aspects of the SystemML internals. The class is scheduled to take place over 8 sessions from June to August, covering topics like regression, classification, clustering, and the SystemML architecture, optimizer, and runtime.
Apache SystemML 2016 Summer class primer by Berthold Reinwald
Apache SystemML 2016 Summer class primer by Berthold ReinwaldArvind Surve This document provides details about an Apache SystemML class being offered in the summer of 2016. The class aims to teach scalable machine learning using Apache SystemML. It will cover SystemML usage, hands-on exercises for developing machine learning algorithms, and advanced aspects of the SystemML internals. The class is scheduled to take place over 8 sessions from June to August, covering topics like regression, classification, clustering, and the SystemML architecture, optimizer, and runtime.
The Data Science Process - Do we need it and how to apply?
The Data Science Process - Do we need it and how to apply?Ivo Andreev The document outlines the Global AI Bootcamp featuring a presentation on machine learning (ML) and its application within Microsoft Azure's services and tools. Key highlights include the Azure ML integrated platform, various machine learning frameworks, challenges of data preparation, and the importance of iterative processes in ML modeling. The document also discusses artificial intelligence as a service (AIaaS) and the anticipated growth of this market, providing insights into how businesses can leverage these technologies for data science and predictive modeling.
Zementis hortonworks-webinar-2014-09
Zementis hortonworks-webinar-2014-09Hortonworks The webinar presented by Hortonworks and Zementis highlights the advantages of Hadoop for machine learning and predictive analytics. Key speakers discussed Hadoop's capabilities in data processing, model deployment, and the integration of PMML for enhanced predictive insights across various industries. The session emphasized the importance of Hadoop in accelerating innovation and achieving business value through effective data management and analysis.
Introduction to Machine Learning - An overview and first step for candidate d...
Introduction to Machine Learning - An overview and first step for candidate d...Lucas Jellema The document provides a comprehensive introduction to machine learning, discussing its relevance, methodologies, and applications, including supervised, unsupervised, and reinforcement learning. It details the data science workflow, tools, and algorithms, emphasizing the importance of big data, open data, and platforms like Oracle and Apache Spark. The summary also highlights the democratization of machine learning and its business implications, encouraging practitioners to explore and adopt these technologies.
Towards a Comprehensive Machine Learning Benchmark
Towards a Comprehensive Machine Learning BenchmarkTuri, Inc. This document presents a framework for developing a comprehensive machine learning benchmark. It discusses identifying the core building blocks of machine learning algorithms, such as linear algebra, data characteristics, and memory access. It proposes evaluating these building blocks using representative algorithms, datasets, and configurations. Thousands of executions are clustered into a smaller set capturing different software and hardware behaviors. The resulting benchmark suite of 50 workloads incorporates the main building blocks and bottlenecks to help evaluate machine learning performance.
[DSC Europe 22] Smart approach in development and deployment process for vari...
[DSC Europe 22] Smart approach in development and deployment process for vari...DataScienceConferenc1 The document outlines a strategic approach for developing and deploying machine learning models, emphasizing the importance of planning throughout the data science lifecycle to reduce production time. It introduces the concept of smart generic data marts to facilitate prototype development, feature engineering, and data quality management while ensuring compliance with regulatory standards. The final goals include the efficient orchestration of production models and the establishment of a model registry for monitoring and collaboration.
The Art of Intelligence – A Practical Introduction Machine Learning for Oracl...
The Art of Intelligence – A Practical Introduction Machine Learning for Oracl...Lucas Jellema The document serves as an introduction to machine learning specifically for Oracle professionals, discussing its relevance, algorithms, and practical applications. It covers the data science workflow, various machine learning types, popular technologies, and how Oracle is advancing in this field. The document concludes with resources and tools for getting started with machine learning.
The Art of Intelligence – Introduction Machine Learning for Oracle profession...
The Art of Intelligence – Introduction Machine Learning for Oracle profession...Lucas Jellema This document serves as an introduction to machine learning, detailing its relevance, algorithms, and the process involved in implementing it. It covers use cases, popular technologies, and how to effectively gather and prepare data for machine learning applications. The author, Lucas Jellema, emphasizes the accessibility and applicability of machine learning across various industries while providing educational resources for beginners.
Prepare your data for machine learning
Prepare your data for machine learningIvo Andreev The document outlines the preparation process for machine learning, focusing on Microsoft's tools and services, such as Azure ML, for data science and advanced analytics. It covers topics like data preparation, model management, feature engineering, and the challenges of machine learning, emphasizing the importance of selecting predictive features and handling imbalanced data. The document also includes pricing information, highlights various tools, and underscores the iterative nature of the data science process.
Apache SystemML Architecture by Niketan Panesar
Apache SystemML Architecture by Niketan PanesarArvind Surve The document outlines the architecture, design, and execution of SystemML, a declarative machine learning language designed to work with Hadoop and Spark. It discusses key components such as APIs, the compilation chain, runtime execution plans, and includes examples of DML expressions and linear regression operations. Additionally, it highlights important links for further exploration and contributions to the SystemML project.
[DSC Europe 22] Smart approach in development and deployment process for vari...
[DSC Europe 22] Smart approach in development and deployment process for vari...DataScienceConferenc1
Ad
More from Jen Aman (19)
Deep Learning and Streaming in Apache Spark 2.x with Matei Zaharia
Deep Learning and Streaming in Apache Spark 2.x with Matei ZahariaJen Aman The document discusses advancements in Apache Spark 2.x, particularly focusing on structured streaming and deep learning capabilities. It highlights the introduction of a high-level API for streaming that simplifies processes and allows for end-to-end applications, as well as the integration of deep learning frameworks to enhance usability. Key features include improved performance, cost efficiency, and the goal of making deep learning accessible to a broader range of users.
Snorkel: Dark Data and Machine Learning with Christopher Ré
Snorkel: Dark Data and Machine Learning with Christopher RéJen Aman The document discusses Snorkel, a project aimed at simplifying machine learning by enabling users to build systems for tasks like data extraction without the need for large hand-labeled training sets. It highlights how Snorkel leverages a method known as data programming, where users create labeling functions to generate noisy training data, enhancing the efficiency of model training. The document emphasizes the potential of machine learning to make programming significantly easier and showcases the effectiveness of Snorkel in achieving competitive results compared to traditional supervised approaches.
Deep Learning on Apache® Spark™: Workflows and Best Practices
Deep Learning on Apache® Spark™: Workflows and Best PracticesJen Aman This document summarizes a presentation about deep learning workflows and best practices on Apache Spark. It discusses how deep learning fits within broader data pipelines for tasks like training and transformation. It also outlines recurring patterns for integrating Spark and deep learning frameworks, including using Spark for data parallelism and embedding deep learning transforms. The presentation provides tips for developers on topics like using GPUs with PySpark and monitoring deep learning jobs. It concludes by discussing challenges in the areas of distributed deep learning and Spark integration.
Deep Learning on Apache® Spark™ : Workflows and Best Practices
Deep Learning on Apache® Spark™ : Workflows and Best PracticesJen Aman The document presents an overview of deep learning workflows and best practices using Apache Spark, highlighting its integration with various deep learning frameworks such as TensorFlow and Caffe. It discusses the architecture of Databricks, which simplifies data processing and machine learning for users, as well as monitoring techniques and solutions for managing distributed deep learning tasks. The presentation emphasizes the importance of experimentation and provides resources for further learning.
Spatial Analysis On Histological Images Using Spark
Spatial Analysis On Histological Images Using SparkJen Aman This document describes using Spark for spatial analysis of histological images to characterize the tumor microenvironment. The goal is to provide actionable data on the location and density of immune cells and blood vessels. Over 100,000 objects are annotated in each whole slide image. Spark is used to efficiently calculate over 5 trillion pairwise distances between objects within a neighborhood window. This enables profiling of co-localization and spatial clustering of objects. Initial results show the runtime scales linearly with the number of objects. Future work includes integrating clinical and genomic data to characterize variation between tumor types and patients.
Massive Simulations In Spark: Distributed Monte Carlo For Global Health Forec...
Massive Simulations In Spark: Distributed Monte Carlo For Global Health Forec...Jen Aman Kyle Foreman presented on using Spark for large-scale global health simulations. The talk discussed (1) the motivation for simulations to model disease burden forecasts and alternative scenarios, (2) SimBuilder for constructing modular simulation workflows as directed acyclic graphs, and (3) benchmarks showing Spark backends can efficiently distribute simulations across a cluster. Future work aims to optimize Spark DataFrame joins and take better advantage of Numpy's vectorization for panel data simulations.
A Graph-Based Method For Cross-Entity Threat Detection
A Graph-Based Method For Cross-Entity Threat DetectionJen Aman This document proposes a graph-based method for cross-entity threat detection. It models entity relationships as a multigraph and detects anomalies by identifying unexpected new connections between entities over time. It introduces two algorithms: a naive detector that identifies edges only in the detection graph, and a 2nd-order detector that identifies edges between entity clusters. An experiment on a real dataset found around 700 1st-order and 200 2nd-order anomalies in under 5 minutes, demonstrating the method's ability to efficiently detect threats across unrelated accounts.
Time-Evolving Graph Processing On Commodity Clusters
Time-Evolving Graph Processing On Commodity ClustersJen Aman Tegra is a system for efficiently processing time-evolving graphs on commodity clusters. It uses a distributed graph snapshot index to represent and retrieve multiple snapshots of evolving graphs. It introduces a timelapse abstraction to perform temporal analytics on windows of snapshots, avoiding redundant computation. Tegra supports both bulk and incremental graph computations using this representation, allowing results to be reused when graphs are updated. An evaluation on real-world graphs shows Tegra can store more snapshots in memory and reduce computation time compared to baseline approaches.
Deploying Accelerators At Datacenter Scale Using Spark
Deploying Accelerators At Datacenter Scale Using SparkJen Aman The document discusses the deployment of accelerators, specifically FPGA and GPU, at datacenter scale to enhance performance and energy efficiency as CPU core scaling nears its limits. It introduces the Blaze system, which provides an accelerator-as-a-service model, optimizing JVM-to-accelerator communication to streamline programming and enhance big data application integration. The paper highlights the challenges in accelerator management and programming, while showcasing improvements in job latency and energy consumption through optimized hardware utilization.
Re-Architecting Spark For Performance Understandability
Re-Architecting Spark For Performance UnderstandabilityJen Aman The document describes a new architecture called "monotasks" for Apache Spark that aims to make reasoning about Spark job performance easier. The monotasks architecture decomposes Spark tasks so that each task uses only one resource (e.g. CPU, disk, network). This avoids issues where Spark tasks bottleneck on different resources over time or experience resource contention. With monotasks, dedicated schedulers control resource contention and monotask timing data can be used to model ideal performance. Results show monotasks match Spark's performance and provide clearer insight into bottlenecks.
Efficient State Management With Spark 2.0 And Scale-Out Databases
Efficient State Management With Spark 2.0 And Scale-Out DatabasesJen Aman This document discusses efficient state management with Spark 2.0 and scale-out databases. It introduces SnappyData, an open source project that provides a unified in-memory database for streams, transactions, and OLAP queries to enable real-time operational analytics. SnappyData extends Spark by localizing state management and processing to avoid shuffles, supports approximate query processing for interactive queries, and provides a unified cluster architecture for OLTP, OLAP and streaming workloads.
GPU Computing With Apache Spark And Python
GPU Computing With Apache Spark And PythonJen Aman GPU Computing With Apache Spark And Python
- Python is a popular language for data science and analytics due to its large ecosystem of libraries and ease of use, but it is slow for number crunching tasks. GPU computing is a way to accelerate Python workloads.
- This presentation demonstrates using GPUs with Apache Spark and Python through libraries like Accelerate, which provides drop-in GPU-accelerated functions, and Numba, which can compile Python functions to run on GPUs.
- As an example, the task of image registration, which involves computationally expensive 2D FFTs, is accelerated using these GPU libraries within a PySpark job, achieving a 2-4x speedup over CPU-only versions
Spark at Bloomberg: Dynamically Composable Analytics
Spark at Bloomberg: Dynamically Composable Analytics Jen Aman The Bloomberg Spark Server provides a platform for dynamic composable analytics on Spark. It includes a Function Transform Registry (FTR) of reusable analytics functions. It also includes a Managed DataFrame Registry (MDFR) that allows request processors to lookup and compose analytics on named DataFrames. The request processors handle analytic requests by looking up functions from the FTR and composing them on MDFs from the MDFR. The Spark Server addresses challenges of standalone Spark apps like data redundancy and cross-asset analytics. It also handles real-time ingestion and memory management to support online analytics on Spark.
EclairJS = Node.Js + Apache Spark
EclairJS = Node.Js + Apache SparkJen Aman EclairJS allows developers to use JavaScript and Node.js to interact with Apache Spark for large-scale data processing and analytics. It provides a Spark API for Node.js so that compute-intensive workloads can be handed off to Spark running in the backend. EclairJS also enables the use of JavaScript with Jupyter notebooks, so data engineers and web developers can experiment with Spark from within the browser using familiar JavaScript syntax.
Spark: Interactive To Production
Spark: Interactive To ProductionJen Aman This document summarizes a presentation given at Spark Summit 2016 about using Spark for real-time data processing and analytics at Uber and Marketplace Data. Some key points:
- Uber generates large amounts of data across its 70+ countries and 450+ cities that is used for real-time processing, analytics, and forecasting.
- Marketplace Data uses Spark for real-time data processing, analytics, and forecasting of Uber's data, which involves challenges like complex event processing, geo aggregation, and querying large and streaming datasets.
- Jupyter notebooks are used to empower users and data scientists to work with Spark in a flexible way, though challenges remain around reliability, freshness, and isolating queries.
High-Performance Python On Spark
High-Performance Python On SparkJen Aman This document contains the slides from a presentation given by Wes McKinney on high performance Python on Apache Spark. The presentation discusses why Python is an important language, defines what is meant by "high performance Python", and explores techniques for building fast Python software such as embracing limitations of the Python interpreter and using native data structures and compiled extensions where possible. Specific examples are provided around control flow, reading CSV files, and the importance of efficient in-memory data structures.
Embrace Sparsity At Web Scale: Apache Spark MLlib Algorithms Optimization For...
Embrace Sparsity At Web Scale: Apache Spark MLlib Algorithms Optimization For...Jen Aman This document discusses optimizations made to Apache Spark MLlib algorithms to better support sparse data at large scale. It describes how KMeans, linear methods, and other ML algorithms were modified to use sparse vector representations to reduce memory usage and improve performance when working with sparse data, including optimizations made for clustering large, high-dimensional datasets. The optimizations allow these algorithms to be applied to much larger sparse datasets and high-dimensional problems than was previously possible with MLlib.
Temporal Operators For Spark Streaming And Its Application For Office365 Serv...
Temporal Operators For Spark Streaming And Its Application For Office365 Serv...Jen Aman This document discusses temporal operators for Spark Streaming and their application for Office365 service monitoring. It introduces temporal operators like reorder, event-time window aggregate, and join that allow normalizing and aggregating streaming data that may arrive out of order. These operators are used to calculate availability metrics over tumbling windows and generate alarms when availability drops between the current and previous windows. Examples show how temporal operators reorder, aggregate, and join streaming events based on timestamps to perform time-based analysis for monitoring Office365 services.
Utilizing Human Data Validation For KPI Analysis And Machine Learning
Utilizing Human Data Validation For KPI Analysis And Machine LearningJen Aman This document discusses utilizing human data validation for key performance indicator (KPI) analysis and machine learning. It describes Radius Intelligence's use of human validation of business data to establish ground truths for training machine learning models and evaluating data sources. While human validation provides benefits, it also incurs costs in money, time, and potential issues between validation teams and data science teams. The document outlines strategies for minimizing validation costs while meeting downstream needs through experimentation and multiple-consumer sampling techniques.
Recently uploaded (20)
Residential Zone 4 for industrial village
Residential Zone 4 for industrial villageMdYasinArafat13 based on assumption that failure of such a weld is by shear on the
effective area whether the shear transfer is parallel to or
perpendicular to the axis of the line of fillet weld. In fact, the
strength is greater for shear transfer perpendicular to the weld axis;
however, for simplicity the situations are treated the same.
最新版美国亚利桑那大学毕业证（UA毕业证书）原版定制
最新版美国亚利桑那大学毕业证（UA毕业证书）原版定制Taqyea 鉴于此，定制亚利桑那大学学位证书提升履历【q薇1954292140】原版高仿亚利桑那大学毕业证(UA毕业证书)可先看成品样本【q薇1954292140】帮您解决在美国亚利桑那大学未毕业难题，美国毕业证购买，美国文凭购买，【q微1954292140】美国文凭购买，美国文凭定制，美国文凭补办。专业在线定制美国大学文凭，定做美国本科文凭，【q微1954292140】复制美国The University of Arizona completion letter。在线快速补办美国本科毕业证、硕士文凭证书，购买美国学位证、亚利桑那大学Offer，美国大学文凭在线购买。
如果您处于以下几种情况：
◇在校期间，因各种原因未能顺利毕业……拿不到官方毕业证
◇面对父母的压力，希望尽快拿到；
◇不清楚认证流程以及材料该如何准备；
◇回国时间很长，忘记办理；
◇回国马上就要找工作，办给用人单位看；
◇企事业单位必须要求办理的
◇需要报考公务员、购买免税车、落转户口
◇申请留学生创业基金
【复刻一套亚利桑那大学毕业证成绩单信封等材料最强攻略,Buy The University of Arizona Transcripts】
购买日韩成绩单、英国大学成绩单、美国大学成绩单、澳洲大学成绩单、加拿大大学成绩单（q微1954292140）新加坡大学成绩单、新西兰大学成绩单、爱尔兰成绩单、西班牙成绩单、德国成绩单。成绩单的意义主要体现在证明学习能力、评估学术背景、展示综合素质、提高录取率，以及是作为留信认证申请材料的一部分。
亚利桑那大学成绩单能够体现您的的学习能力，包括亚利桑那大学课程成绩、专业能力、研究能力。（q微1954292140）具体来说，成绩报告单通常包含学生的学习技能与习惯、各科成绩以及老师评语等部分，因此，成绩单不仅是学生学术能力的证明，也是评估学生是否适合某个教育项目的重要依据！
最新版美国芝加哥大学毕业证（UChicago毕业证书）原版定制
最新版美国芝加哥大学毕业证（UChicago毕业证书）原版定制taqyea 2025原版芝加哥大学毕业证书pdf电子版【q薇1954292140】美国毕业证办理UChicago芝加哥大学毕业证书多少钱？【q薇1954292140】海外各大学Diploma版本，因为疫情学校推迟发放证书、证书原件丢失补办、没有正常毕业未能认证学历面临就业提供解决办法。当遭遇挂科、旷课导致无法修满学分，或者直接被学校退学，最后无法毕业拿不到毕业证。此时的你一定手足无措，因为留学一场，没有获得毕业证以及学历证明肯定是无法给自己和父母一个交代的。
【复刻芝加哥大学成绩单信封,Buy The University of Chicago Transcripts】
购买日韩成绩单、英国大学成绩单、美国大学成绩单、澳洲大学成绩单、加拿大大学成绩单（q微1954292140）新加坡大学成绩单、新西兰大学成绩单、爱尔兰成绩单、西班牙成绩单、德国成绩单。成绩单的意义主要体现在证明学习能力、评估学术背景、展示综合素质、提高录取率，以及是作为留信认证申请材料的一部分。
芝加哥大学成绩单能够体现您的的学习能力，包括芝加哥大学课程成绩、专业能力、研究能力。（q微1954292140）具体来说，成绩报告单通常包含学生的学习技能与习惯、各科成绩以及老师评语等部分，因此，成绩单不仅是学生学术能力的证明，也是评估学生是否适合某个教育项目的重要依据！
我们承诺采用的是学校原版纸张（原版纸质、底色、纹路）我们工厂拥有全套进口原装设备，特殊工艺都是采用不同机器制作，仿真度基本可以达到100%，所有成品以及工艺效果都可提前给客户展示，不满意可以根据客户要求进行调整，直到满意为止！
【主营项目】
一、工作未确定，回国需先给父母、亲戚朋友看下文凭的情况，办理毕业证|办理文凭: 买大学毕业证|买大学文凭【q薇1954292140】芝加哥大学学位证明书如何办理申请？
二、回国进私企、外企、自己做生意的情况，这些单位是不查询毕业证真伪的，而且国内没有渠道去查询国外文凭的真假，也不需要提供真实教育部认证。鉴于此，办理美国成绩单芝加哥大学毕业证【q薇1954292140】国外大学毕业证, 文凭办理, 国外文凭办理, 留信网认证
Power BI API Connectors - Best Practices for Scalable Data Connections
Power BI API Connectors - Best Practices for Scalable Data Connections Vidicorp Ltd This document explains how API connections work in Power BI, comparing different methods like custom connectors, push datasets, database pipelines, and third-party integration tools. It also gives a clear plan on how to use APIs for flexible, scalable, and smart business reporting.
最新版美国威斯康星大学拉克罗斯分校毕业证（UW–L毕业证书）原版定制
最新版美国威斯康星大学拉克罗斯分校毕业证（UW–L毕业证书）原版定制Taqyea 鉴于此，定制威斯康星大学拉克罗斯分校学位证书提升履历【q薇1954292140】原版高仿威斯康星大学拉克罗斯分校毕业证(UW–L毕业证书)可先看成品样本【q薇1954292140】帮您解决在美国威斯康星大学拉克罗斯分校未毕业难题，美国毕业证购买，美国文凭购买，【q微1954292140】美国文凭购买，美国文凭定制，美国文凭补办。专业在线定制美国大学文凭，定做美国本科文凭，【q微1954292140】复制美国University of Wisconsin-La Crosse completion letter。在线快速补办美国本科毕业证、硕士文凭证书，购买美国学位证、威斯康星大学拉克罗斯分校Offer，美国大学文凭在线购买。
如果您处于以下几种情况：
◇在校期间，因各种原因未能顺利毕业……拿不到官方毕业证
◇面对父母的压力，希望尽快拿到；
◇不清楚认证流程以及材料该如何准备；
◇回国时间很长，忘记办理；
◇回国马上就要找工作，办给用人单位看；
◇企事业单位必须要求办理的
◇需要报考公务员、购买免税车、落转户口
◇申请留学生创业基金
【复刻一套威斯康星大学拉克罗斯分校毕业证成绩单信封等材料最强攻略,Buy University of Wisconsin-La Crosse Transcripts】
购买日韩成绩单、英国大学成绩单、美国大学成绩单、澳洲大学成绩单、加拿大大学成绩单（q微1954292140）新加坡大学成绩单、新西兰大学成绩单、爱尔兰成绩单、西班牙成绩单、德国成绩单。成绩单的意义主要体现在证明学习能力、评估学术背景、展示综合素质、提高录取率，以及是作为留信认证申请材料的一部分。
威斯康星大学拉克罗斯分校成绩单能够体现您的的学习能力，包括威斯康星大学拉克罗斯分校课程成绩、专业能力、研究能力。（q微1954292140）具体来说，成绩报告单通常包含学生的学习技能与习惯、各科成绩以及老师评语等部分，因此，成绩单不仅是学生学术能力的证明，也是评估学生是否适合某个教育项目的重要依据！
Managed Cloud services - Opsio Cloud Man
Managed Cloud services - Opsio Cloud ManOpsio Cloud At Opsio, we specialize in delivering advanced cloud services that enable businesses to scale, transform, and modernize with confidence. Our core offerings focus on cloud management, digital transformation, and cloud modernization — all designed to help organizations unlock the full potential of their technology infrastructure.We take a client-first approach, blending industry-leading hosted technologies with strategic expertise to create tailored, future-ready solutions. Leveraging AI, automation, and emerging technologies, our services simplify IT operations, enhance agility, and accelerate business outcomes. Whether you're migrating to the cloud or optimizing existing cloud environments, Opsio is your partner in achieving sustainable, measurable success.
apidays Singapore 2025 - Building Finance Innovation Ecosystems by Umang Moon...
apidays Singapore 2025 - Building Finance Innovation Ecosystems by Umang Moon...apidays Building Finance Innovation Ecosystems
Umang Moondra, CEO at APIX
apidays Singapore 2025
Where APIs Meet AI: Building Tomorrow's Intelligent Ecosystems
April 15 & 16, 2025
------
Check out our conferences at https://www.apidays.global/
Do you want to sponsor or talk at one of our conferences?
https://apidays.typeform.com/to/ILJeAaV8
Learn more on APIscene, the global media made by the community for the community:
https://www.apiscene.io
Explore the API ecosystem with the API Landscape:
https://apilandscape.apiscene.io/
apidays New York 2025 - API Security and Observability at Scale in Kubernetes...
apidays New York 2025 - API Security and Observability at Scale in Kubernetes...apidays API Security and Observability at Scale in Kubernetes
Ben Urbanski, Product Manager for Layer7 at Broadcom
Geoffrey Duck, Solution Engineer and Service Lead for Layer7 at Broadcom
apidays New York 2025
API Management for Surfing the Next Innovation Waves: GenAI and Open Banking
May 14 & 15, 2025
------
Check out our conferences at https://www.apidays.global/
Do you want to sponsor or talk at one of our conferences?
https://apidays.typeform.com/to/ILJeAaV8
Learn more on APIscene, the global media made by the community for the community:
https://www.apiscene.io
Explore the API ecosystem with the API Landscape:
https://apilandscape.apiscene.io/
What is FinOps as a Service and why is it Trending?
What is FinOps as a Service and why is it Trending?Amnic The way we build and scale companies today has changed forever because of cloud adoption. However, this flexibility introduces unpredictability, which often results in overspending, inefficiencies, and a lack of cost accountability.
FinOps as a Service is a modern approach to cloud cost management that combines powerful tooling with expert advisory to bring financial visibility, governance, and optimization into the cloud operating model, without slowing down the engineering team. FinOps empowers the engineering team, finance, and leadership/management as they make data-informed decisions about cost, together.
In this presentation, we will break down what FinOps is, why it matters more than ever, and a little about how a managed FinOps service can help organizations:
- Optimize cloud spend - without slowing down dev
- Create visibility into the cost per team, service, or feature
- Set financial guardrails while allowing autonomy in engineering
- Drive cultural alignment between finance, engineering, and product
This will guide and help whether you are a cloud-native startup or a scaling enterprise, and convert cloud cost into a strategic advantage.
Fundamental Analysis for Dummies.pdf somwmdw
Fundamental Analysis for Dummies.pdf somwmdwssuserc74044 axanxbajbjbjxxxxxxxxxxxx AHXBAHVBXHVVVVVVVVVVVVVV AASJAHSJAHSJA ASXHSKAXKAXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXAXANNN ZNXSBXJSBJX AIXHIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIA AXHIHHHHHHHHHHHHHHHHHHHHHHHHH AIXHIIIIIIIIIIIIIIIIIIIIIIIIIIII IAHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHHH AXHIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIIII SOMEM AXSKXJSKXJSKJASKJOSJCIUDHDUVBDJBN SICJIOSS HSSSSSSSSSSSSSSSSCBBBBBBBBBBBBBBBBSNCBSJCBSJCBSJCBSJCBSJBCJSBCJS XXXBHBXSBHJSB ONE DAY XNSBNSBXSJS SCSSJKSJ SOME TIWDSCNSCN S
Data Warehousing and Analytics IFI Techsolutions .pptx
Data Warehousing and Analytics IFI Techsolutions .pptxIFI Techsolutions IFI Techsolutions is a leading cloud solutions and managed services provider. We are an Azure Expert Managed Service Provider (AEMSP) and boast multiple advanced specializations in Infra and Database Migration to Microsoft Azure, Migrate Enterprise Apps to Microsoft Azure, and Microsoft Azure Virtual Desktop.
Simplify SQL migration to Azure with Microsoft’s DMA tool—assess, plan, and move your data seamlessly with expert support from IFI Tech.
KLIP2Data voor de herinrichting van R4 West en Oost
KLIP2Data voor de herinrichting van R4 West en Oostjacoba18 Ontwikkeling van een centrale GIS-databank voor de periodieke import van KLIP-data, BIM- en CAD-modellen van het R4-project en bijhorende field app om de databank zowel online als offline te raadplegen en bij te werken.
YEAP !NOT WHAT YOU THINK aakshdjdncnkenfj
YEAP !NOT WHAT YOU THINK aakshdjdncnkenfjpayalmistryb Skdnneodnrodndocndodnd0dndjxoxnxndkxnxkdndkxndkdjndnrnidnroz doendodnrodnxkdnrocngksjrndkdnr dnkxnddnkxndnrkdnxnxkdSkdnneodnrodndocndodnd0dndjxoxnxndkxnxkdndkxndkdjndnrnidnroz doendodnrodnxkdnrocngksjrndkdnr dnkxnddnkxndnrkdnxnxkdSkdnneodnrodndocndodnd0dndjxoxnxndkxnxkdndkxndkdjndnrnidnroz doendodnrodnxkdnrocngksjrndkdnr dnkxnddnkxndnrkdnxnxkdSkdnneodnrodndocndodnd0dndjxoxnxndkxnxkdndkxndkdjndnrnidnroz doendodnrodnxkdnrocngksjrndkdnr dnkxnddnkxndnrkdnxnxkdSkdnneodnrodndocndodnd0dndjxoxnxndkxnxkdndkxndkdjndnrnidnroz doendodnrodnxkdnrocngksjrndkdnr dnkxnddnkxndnrkdnxnxkdSkdnneodnrodndocndodnd0dndjxoxnxndkxnxkdndkxndkdjndnrnidnroz doendodnrodnxkdnrocngksjrndkdnr dnkxnddnkxndnrkdnxnxkdSkdnneodnrodndocndodnd0dndjxoxnxndkxnxkdndkxndkdjndnrnidnroz doendodnrodnxkdnrocngksjrndkdnr dnkxnddnkxndnrkdnxnxkdSkdnneodnrodndocndodnd0dndjxoxnxndkxnxkdndkxndkdjndnrnidnroz doendodnrodnxkdnrocngksjrndkdnr dnkxnddnkxndnrkdnxnxkdSkdnneodnrodndocndodnd0dndjxoxnxndkxnxkdndkxndkdjndnrnidnroz doendodnrodnxkdnrocngksjrndkdnr dnkxnddnkxndnrkdnxnxkdSkdnneodnrodndocndodnd0dndjxoxnxndkxnxkdndkxndkdjndnrnidnroz doendodnrodnxkdnrocngksjrndkdnr dnkxnddnkxndnrkdnxnxkdSkdnneodnrodndocndodnd0dndjxoxnxndkxnxkdndkxndkdjndnrnidnroz doendodnrodnxkdnrocngksjrndkdnr dnkxnddnkxndnrkdnxnxkdSkdnneodnrodndocndodnd0dndjxoxnxndkxnxkdndkxndkdjndnrnidnroz doendodnrodnxkdnrocngksjrndkdnr dnkxnddnkxndnrkdnxnxkdSkdnneodnrodndocndodnd0dndjxoxnxndkxnxkdndkxndkdjndnrnidnroz doendodnrodnxkdnrocngksjrndkdnr dnkxnddnkxndnrkdnxnxkdSkdnneodnrodndocndodnd0dndjxoxnxndkxnxkdndkxndkdjndnrnidnroz doendodnrodnxkdnrocngksjrndkdnr dnkxnddnkxndnrkdnxnxkdSkdnneodnrodndocndodnd0dndjxoxnxndkxnxkdndkxndkdjndnrnidnroz doendodnrodnxkdnrocngksjrndkdnr dnkxnddnkxndnrkdnxnxkdSkdnneodnrodndocndodnd0dndjxoxnxndkxnxkdndkxndkdjndnrnidnroz doendodnrodnxkdnrocngksjrndkdnr dnkxnddnkxndnrkdnxnxkdSkdnneodnrodndocndodnd0dndjxoxnxndkxnxkdndkxndkdjndnrnidnroz doendodnrodnxkdnrocngksjrndkdnr dnkxnddnkxndnrkdnxnxkdSkdnneodnrodndocndodnd0dndjxoxnxndkxnxkdndkxndkdjndnrnidnroz doendodnrodnxkdnrocngksjrndkdnr dnkxnddnkxndnrkdnxnxkdSkdnneodnrodndocndodnd0dndjxoxnxndkxnxkdndkxndkdjndnrnidnroz doendodnrodnxkdnrocngksjrndkdnr dnkxnddnkxndnrkdnxnxkdSkdnneodnrodndocndodnd0dndjxoxnxndkxnxkdndkxndkdjndnrnidnroz doendodnrodnxkdnrocngksjrndkdnr dnkxnddnkxndnrkdnxnxkdSkdnneodnrodndocndodnd0dndjxoxnxndkxnxkdndkxndkdjndnrnidnroz doendodnrodnxkdnrocngksjrndkdnr dnkxnddnkxndnrkdnxnxkdSkdnneodnrodndocndodnd0dndjxoxnxndkxnxkdndkxndkdjndnrnidnroz doendodnrodnxkdnrocngksjrndkdnr dnkxnddnkxndnrkdnxnxkdSkdnneodnrodndocndodnd0dndjxoxnxndkxnxkdndkxndkdjndnrnidnroz doendodnrodnxkdnrocngksjrndkdnr dnkxnddnkxndnrkdnxnxkdSkdnneodnrodndocndodnd0dndjxoxnxndkxnxkdndkxndkdjndnrnidnroz doendodnrodnxkdnrocngksjrndkdnr dnkxnddnkxndnrkdnxnxkdSkdnneodnrodndocndodnd0dndjxoxnxndkxnxkdndkxndkdjndnrnidnroz doendodnrodnxkdnrocngksjrndkdnr dnkxnddnkxndnrkdnxnxkdSkdnneodnrodndocndodnd0dndjxoxnxndkxnxkdndkxndkdjndnrnidnroz doendodnrodnxkdnrocngksjrndkdnr dnkxnddnkxndnrkdnxnxkdSkdnne
apidays Singapore 2025 - 4 Identity Essentials for Scaling SaaS in Large Orgs...
apidays Singapore 2025 - 4 Identity Essentials for Scaling SaaS in Large Orgs...apidays 4 identity factors you didn't know you needed to support large organizations in your SaaS
Daizen Ikehara, Principal Developer Advocate at Auth0
apidays Singapore 2025
Where APIs Meet AI: Building Tomorrow's Intelligent Ecosystems
April 15 & 16, 2025
------
Check out our conferences at https://www.apidays.global/
Do you want to sponsor or talk at one of our conferences?
https://apidays.typeform.com/to/ILJeAaV8
Learn more on APIscene, the global media made by the community for the community:
https://www.apiscene.io
Explore the API ecosystem with the API Landscape:
https://apilandscape.apiscene.io/
METHODS OF DATA COLLECTION (Research methodology)
METHODS OF DATA COLLECTION (Research methodology)anwesha248 The important methods of data collection along with advantages and disadvantages are published in this power point presentation.
published by- Prateeti Anwesha Bordoloi, Research Scholar
Mahapurusha Srimanta Sankaradeva Vishwavidyalaya
June 2025
Media_Literacy_Index_of_Media_Sector_Employees.pdf
Media_Literacy_Index_of_Media_Sector_Employees.pdfOlhaTatokhina1 Report based on the findings of a quantitative research conducted by the research agency New Image Marketing Group, commissioned by the NGO Detector Media, compiled by PhD in Sociology Marta Naumova.
Building Custom Machine Learning Algorithms With Apache SystemML
- 1. Building Custom Machine Learning Algorithms with Apache SystemML Fred Reiss Chief Architect, IBM Spark Technology Center Member, IBM Academy of Technology
- 2. Roadmap • What is Apache SystemML? • Demo! • How to get SystemML
- 3. What is Apache SystemML?
- 4. Origins of the SystemML Project 20162015 You are here.
- 6. 200920082007 2007-2008: Multiple projects at IBM Research – Almaden involving machine learning on Hadoop. 2010 2009-2010: Through engagements with customers, we observe how data scientists create ML solutions. 2009: We form a dedicated team for scalable ML
- 7. Case Study: An Auto Manufacturer Warranty Claims Repair History Diagnostic Readouts Predict Reacquired Cars
- 8. Case Study: An Auto Manufacturer Warranty Claims Repair History Features Labels Predict Reacquired Cars Machine Learning Algorithm Algorithm Algorithm Algorithm Result: 25x improvement in precision! False Positives Diagnostic Readouts
- 9. The Iterative Development Process Build a pipeline Results good enough? Yes Customize part of the pipeline No
- 10. State-of-the-Art: Small Data R or Python Data Scientist Personal Computer Data Results
- 11. State-of-the-Art: Big Data R or Python Data Scientist Results Systems Programmer Scala
- 12. State-of-the-Art: Big Data R or Python Data Scientist Results Systems Programmer Scala 😞 Days or weeks per iteration 😞 Errors while translating algorithms
- 13. The SystemML Vision R or Python Data Scientist Results SystemML
- 14. The SystemML Vision R or Python Data Scientist Results SystemML 😃 Fast iteration 😃 Same answer
- 15. 200920082007 2007-2008: Multiple projects at IBM Research – Almaden involving machine learning on Hadoop. 2010 2009-2010: Through engagements with customers, we observe how data scientists create machine learning algorithms. 2009: We form a dedicated team for scalable ML
- 17. 20162015 Apache SystemML June 2015: IBM Announces open- source SystemML September 2015: Code available on Github November 2015: SystemML enters Apache incubation June 2016: Second Apache release (0.10) February 2016: First release (0.9) of Apache SystemML
- 18. SystemML at • Built algorithms for predicting treatment outcomes – Substantial improvement in accuracy • Moved from Hadoop MapReduce to Spark – SystemML supports both frameworks – Exact same code – 300X faster on 1/40th as many nodes
- 19. SystemML at Cadent Technology “SystemML allows Cadent to implement advanced numerical programming methods in Apache Spark, empowering us to leverage specialized algorithms in our predictive analytics software.” Michael Zargham Chief Scientist Cadent is a leading provider of TV advertising and data solutions, reaching over 140 million homes and trusted by the world’s largest service providers.
- 20. Demo!
- 21. Demo Scenario • Application: Targeted ads using demographic information tied to cookies • Problem: The information is incomplete • Solution: Estimate the missing values – Treat the problem as a matrix completion problem
- 22. Data • The U.S. Census Public Use Microdata Sample (PUMS) data set for 2010 • 10% sample of the U.S. population – We’ll use just California today • Use this full data set to generate synthetic incomplete data
- 23. Demo Scenario • Application: Identify products that are complementary (often purchased together) • Problem: Customers are not currently buying the best complements at the same time • Solution: Suggest new product pairings – Treat the problem as a matrix completion problem
- 24. Demographics Users i j Value of demographic field j for customer i Matrix Factorization Top Factor LeftFactor Multiply these two factors to produce a less- sparse matrix. × New nonzero values become interpolated demographic information
- 25. Demo Part 1: Data wrangling
- 26. Demo Part 2: Custom algorithm
- 27. Key Points • SystemML, Spark, and Zeppelin work together • Linear algebra is great for data science • Customization is important
- 28. How to get Apache SystemML
- 29. The Apache SystemML Web Site http://systemml.apache.org Download the binary release! Try out some tutorials! Browse the source! Contribute to the project!
- 30. THANK YOU. Please try out Apache SystemML! http://systemml.apache.org Special thanks to Nakul Jindal and Mike Dusenberry for helping with the demo!