Scaling Analytics with Apache Spark
1 like1,191 views
The document outlines a meetup hosted by Quantuniversity on August 8, 2016, focusing on scaling analytics using Apache Spark. It highlights the features and advantages of Apache Spark, such as its speed, ease of use, and integration capabilities, along with examples of its applications in machine learning and regression. Additionally, it mentions upcoming training programs and events related to data analytics and Spark workshops.
1 of 43
Downloaded 32 times
Ad
Recommended
Credit risk meetup
Credit risk meetupQuantUniversity This document summarizes a presentation given by Sri Krishnamurthy on machine learning applications for credit risk. The presentation covered topics such as credit scoring models, traditional credit assessment approaches, peer-to-peer lending, and using clustering algorithms and dimensionality reduction techniques to analyze lending club loan data. Specific clustering and visualization methods discussed include partitioning around medoids, t-distributed stochastic neighborhood embedding, and analyzing mixed data types using Gower similarity coefficients.
Outlier analysis for Temporal Datasets
Outlier analysis for Temporal DatasetsQuantUniversity The document presents details about a QuantUniversity meetup focused on anomaly detection for temporal datasets, held in Boston on July 11, 2016. It outlines the agenda, including training sessions and workshops on anomaly detection methods, and discusses various techniques for identifying outliers in data using statistical, graphical, and machine learning methods. The document also highlights the founder's background and upcoming events related to analytics training and workshops.
Anomaly detection Meetup Slides
Anomaly detection Meetup SlidesQuantUniversity The document outlines a meetup focusing on anomaly detection techniques presented by Sri Krishnamurthy, founder of Quantuniversity LLC, detailing various methods including graphical, statistical, and machine learning approaches. It emphasizes the importance of identifying outliers in large datasets across fields such as finance and energy, along with key upcoming events and training programs from Quantuniversity. Additional resources and examples illustrate methodologies like boxplots, statistical testing, and clustering for detecting anomalies.
Anomaly detection: Core Techniques and Advances in Big Data and Deep Learning
Anomaly detection: Core Techniques and Advances in Big Data and Deep LearningQuantUniversity The document details a meetup hosted by Quantuniversity on anomaly detection techniques, emphasizing the significance of outlier detection in various fields such as finance and energy. It presents methodologies including graphical, statistical, and machine learning approaches to identify anomalies, along with specific examples and workshops offered for training. The founder, Sri Krishnamurthy, also highlights upcoming certification programs and related resources available for further learning.
Anomaly detection
Anomaly detectionQuantUniversity The document discusses anomaly detection techniques and best practices, including its definition and importance in various fields like fraud detection, medical anomalies, and intrusion detection. It outlines several methods for detecting anomalies, such as graphical approaches (boxplots, scatter plots), statistical tests (chi-square and Grubb's test), and machine learning methods (autoencoders, PCA, and clustering). The significance of these techniques is underscored in the context of big data and their increasing application in diverse industries.
Machine learning meetup
Machine learning meetupQuantUniversity The document outlines a presentation by Sri Krishnamurthy at a QuantUniversity meetup in Boston on machine learning basics, focusing on analytics education and upcoming training programs. It covers essential topics such as data types, descriptive and predictive analytics, machine learning algorithms, and evaluation methods. Additionally, it promotes various workshops and courses related to analytics, starting in September 2016.
Anomaly detection : QuantUniversity Workshop
Anomaly detection : QuantUniversity Workshop QuantUniversity The document presents a detailed overview of anomaly detection techniques in data analytics, covering definitions, classifications, applications, and methodologies such as graphical approaches, statistical testing, and machine learning. It provides insights into practical implementations using R programming, highlighting various methods like point, contextual, and collective anomalies, alongside specific examples in fields like fraud detection and medical diagnostics. The content also discusses tools and libraries available for executing these techniques effectively.
Optimization
OptimizationQuantUniversity The document outlines the importance of optimization in various fields, especially in quantitative finance and machine learning, highlighting how it is integral to tasks like portfolio optimization and algorithmic trading. It discusses the different types of optimization problems, such as convex and non-convex, and introduces techniques like steepest descent and stochastic gradient descent. Additionally, it emphasizes the role of optimization in enhancing model performance and the ongoing challenges and tools available in the domain.
Ds for finance day 2
Ds for finance day 2QuantUniversity This document outlines the agenda and content for Day 2 of a 4-day course on Data Science for Finance. Day 2 focuses on building machine learning models, including supervised and unsupervised algorithms. The day consists of four modules: exploring data sets in Python, building a first model using credit risk data, evaluating models using different metrics, and deploying models for prediction. Key machine learning concepts that will be covered include linear regression, logistic regression, neural networks, K-means clustering, and anomaly detection. Labs allow attendees to apply these techniques to a credit risk case study.
Deep learning QuantUniversity meetup
Deep learning QuantUniversity meetupQuantUniversity This document summarizes a deep learning tutorial presentation given by Sri Krishnamurthy. The presentation introduced machine learning and deep learning concepts. It provided demonstrations of deep learning techniques like convolutional neural networks using Python libraries like Theano and Keras. Applications discussed included image recognition, speech recognition, question answering and cybersecurity. Recurrent neural networks and autoencoders were also introduced. The presenter emphasized the role of data, hardware and new approaches in advancing deep learning research.
Nlp and Neural Networks workshop
Nlp and Neural Networks workshopQuantUniversity The document is a comprehensive primer on neural network models specifically tailored for natural language processing (NLP), covering foundational concepts, applications, and various architectures such as recurrent and convolutional neural networks. It discusses the importance of understanding language, the challenges presented by ambiguity and context, and outlines applications in fields like finance and sentiment analysis. Additionally, it highlights training methods and the use of embeddings to improve word representation in NLP tasks.
An Introduction to Anomaly Detection
An Introduction to Anomaly DetectionKenneth Graham The document provides a comprehensive overview of anomaly detection, explaining its definition, types of anomalies, and various detection methods such as supervised, semi-supervised, and unsupervised techniques. It highlights the importance of understanding data context, types of input data, and relationships between instances for effective anomaly detection in applications like credit card fraud and medical data. Additionally, it discusses the use of neural networks, statistical methods, and graph-based techniques for identifying different kinds of anomalies.
Anomaly Detection for Real-World Systems
Anomaly Detection for Real-World SystemsManojit Nandi (1) Anomaly detection aims to identify data points that are noticeably different from expected patterns in a dataset. (2) Common approaches include statistical modeling, machine learning classification, and algorithms designed specifically for anomaly detection. (3) Streaming data poses unique challenges due to limited memory and need for rapid identification of anomalies. (4) Heuristics like z-scores and median absolute deviation provide robust ways to measure how extreme observations are compared to a distribution's center. (5) Density-based methods quantify how isolated data points are to identify anomalies. (6) Time series algorithms decompose trends and seasonality to identify global and local anomalous spikes and troughs.
Feature Selection for Document Ranking
Feature Selection for Document RankingAndrea Gigli The document discusses feature selection algorithms for learning to rank, focusing on proposed feature selection protocols and their application to publicly available web search data. It outlines various competing algorithms including naive greedy search and hierarchical clustering, detailing methods to measure feature relevance and selection performance. Additionally, it describes machine learning strategies for ranking and provides insights into the operational mechanics of ranking systems and their evaluation metrics.
Heuristic design of experiments w meta gradient search
Heuristic design of experiments w meta gradient searchGreg Makowski The document discusses heuristic design techniques for experimenting with model training parameters in data mining, aiming to help data miners navigate a plethora of algorithms and choices while maintaining organization and planning. It emphasizes the importance of structured methodologies, such as meta-gradient search, and the use of a model notebook to track and evaluate model performance effectively. Key concepts include regression testing, decision trees, and neural networks, along with strategies to minimize volatility and enhance reliability in model deployment.
Musings of kaggler
Musings of kagglerKai Xin Thia The document discusses the author's approach to a Kaggle competition by first preparing the data. Key steps included identifying outliers using median absolute deviation, using a Kolmogorov-Smirnov test to identify different distributions in the training data, and removing data that did not need to be modeled. The author then created various features from the data and experimented with different models before creating an ensemble of models. The ensemble approach combined predictions from random forests, gradient boosted models, support vector machines, and generalized linear models through a stacking regression. This process resulted in a top ranking of 49 out of 532 for the competition.
Biostatistics Workshop: Missing Data
Biostatistics Workshop: Missing DataHopkinsCFAR This document discusses approaches for handling missing data in statistical analyses. It begins by contrasting an ideal scenario with no missing data against the real-world problem of missing data. Common types of missing data are defined, including missing completely at random (MCAR), missing at random (MAR), and missing not at random (MNAR). Complete case analysis is described as biased except for large samples with little missing data. Alternative approaches like multiple imputation are presented along with their advantages and disadvantages. An example using indicator variables to handle missing clinic data in a regression is provided. Issues like creating indicators for each missing covariate are also noted.
Machine Learning Algorithm & Anomaly detection 2021
Machine Learning Algorithm & Anomaly detection 2021Chakrit Phain This document discusses various techniques for anomaly detection using machine learning, including:
1. It introduces machine learning and anomaly detection.
2. It covers different machine learning techniques for anomaly detection like genetic algorithms, Monte Carlo simulation, reinforcement learning, and generative adversarial networks.
3. It describes different types of anomalies like point, contextual, and collective anomalies. It also discusses techniques for identifying outliers.
4. Specific anomaly detection techniques are discussed, including recurrent neural networks, historical analysis with DBSCAN clustering, time shift detection, and text pattern anomaly detection using techniques like cosine similarity and autoencoders.
Missing Data and data imputation techniques
Missing Data and data imputation techniquesOmar F. Althuwaynee The document discusses techniques for imputing missing data (<NA>) in R. It introduces common imputation methods like MICE, missForest, and Hmisc. MICE creates multiple imputations using chained equations to account for uncertainty, while missForest uses random forests to impute missing values. Hmisc offers functions to impute missing values using methods like mean, regression, and predictive mean matching. The goal is to understand missing data, learn imputation methods, and choose the best approach for a given dataset.
Explainable AI Workshop
Explainable AI WorkshopQuantUniversity This document presents a workshop on a unified framework for model explanation, focusing on the concept of feature removal in Explainable AI (XAI). It outlines various methods and challenges in the field, emphasizing the need for a coherent theory to guide practitioners and researchers in effectively interpreting machine learning models. The talk concludes with the importance of counterfactual reasoning and the potential trade-offs in choosing appropriate methods for model explanation.
Machine Learning for Dummies
Machine Learning for DummiesVenkata Reddy Konasani This document provides an introduction to machine learning, including:
- It discusses how the human brain learns to classify images and how machine learning systems are programmed to perform similar tasks.
- It provides an example of image classification using machine learning and discusses how machines are trained on sample data and then used to classify new queries.
- It outlines some common applications of machine learning in areas like banking, biomedicine, and computer/internet applications. It also discusses popular machine learning algorithms like Bayes networks, artificial neural networks, PCA, SVM classification, and K-means clustering.
Imputation Techniques For Market Research Datasets With Missing Values
Imputation Techniques For Market Research Datasets With Missing Values Salford Systems This document discusses techniques for handling missing data in statistical analysis and modeling. It compares different modeling approaches on three datasets - one on shoe preferences from a stated preference survey, one on diabetes risk factors, and one on homeowner characteristics. It finds that classification and regression tree (CART) and multivariate adaptive regression splines (MARS) techniques are preferred for imputing missing values when the data contains mixed variable types and interactions among variables. CART can sequentially impute missing values for each variable while preserving the multivariate structure of the data.
Linear regression on 1 terabytes of data? Some crazy observations and actions
Linear regression on 1 terabytes of data? Some crazy observations and actionsHesen Peng 1) The document discusses using linear regression on 1 terabyte of data by leveraging Amazon Web Services' free tier and distributed computing algorithms in Python and R.
2) It notes the challenges of going beyond linear models with big data, including better prediction and real-time analytics.
3) A proposed solution is "universal association discovery" to find relationships between random variables regardless of form using functions on observation graphs, though this approach currently only works for continuous variables.
Aggregating Multiple Dimensions for Computing Document Relevance
Aggregating Multiple Dimensions for Computing Document RelevanceJosé Ramón Ríos Viqueira The document discusses a multi-criteria framework for evaluating document relevance, emphasizing the importance of various factors such as user profiles, document content, and specific criteria in retrieving relevant documents. It presents multiple case studies that demonstrate methods for combining and scoring documents based on different dimensions, including textual and metadata layers. The document ultimately aims to enhance information retrieval systems by adopting more nuanced and personalized approaches to scoring and relevance assessment.
udacity-dandsyllabus
udacity-dandsyllabusBora Yüret This document provides a syllabus for the Data Analyst Nanodegree program. It outlines 7 projects that students will complete to learn data analysis skills in Python, R, SQL, and MongoDB. The projects involve analyzing San Francisco bike share data, testing psychological effects, investigating datasets, wrangling OpenStreetMap data, exploring and summarizing data in R, introductory machine learning to detect fraud, and designing effective data visualizations. Supporting lessons teach statistical techniques, data wrangling, data visualization, and A/B testing.
Andrew Bossy. Data Imputation Using Reverse ML
Andrew Bossy. Data Imputation Using Reverse MLLviv Startup Club This document discusses various methods for data imputation and restoration using reverse machine learning. It describes types of missing data and common imputation techniques like mean, median, mode, regression, KNN, and MICE. Advanced generative models for imputation including VAE, GAIN, MisGAN, and VIGAN are also covered. The document proposes training autoencoders with dropout and regularization as a reverse machine learning technique for data restoration, finding it achieves better accuracy than classical methods.
Statistical Approaches to Missing Data
Statistical Approaches to Missing DataDataCards The document discusses the challenges of missing data in structured datasets, defining types of missingness (MCAR, MAR, and MNAR) and their implications for statistical analysis. It introduces various methods for handling missing data, such as imputation, interpolation, and the need for appropriate techniques depending on the nature of the missing data. The importance of understanding the mechanisms of missingness and their impact on data analysis outcomes is emphasized.
Missing Data and Causes
Missing Data and Causesakanni azeez olamide Missing data occurs when no data value is stored for a variable in an observation, usually due to manual errors or incorrect measurements. There are three types of missing data: missing completely at random, missing at random, and missing not at random. Several methods can be used to deal with missing data, including reducing the dataset, treating missing values as a special value, replacing with the mean, replacing with the most common value, and using the closest fit to impute missing values. Proper handling of missing data is important to avoid bias and distortions in analyzing the data.
Deep learning Tutorial - Part II
Deep learning Tutorial - Part IIQuantUniversity The document is a presentation by Sri Krishnamurthy at a Quantuniversity meetup, covering various aspects of deep learning, specifically focusing on neural networks, autoencoders, and recurrent neural networks (RNNs). It outlines upcoming workshops, provides a recap of concepts like supervised and unsupervised learning, and discusses practical applications of neural networks in areas such as sentiment analysis and forecasting. The presentation emphasizes the significance of tools like Keras for implementing deep learning techniques.
Deep learning - Part I
Deep learning - Part IQuantUniversity The document presents an introduction to deep learning during a meetup hosted by Quantuniversity, highlighting the significance and applications of neural networks. Sri Krishnamurthy, the founder of Quantuniversity, discusses various tools and techniques in analytics, including Keras and Theano, and outlines future events related to deep learning education. It emphasizes the evolution and potential of deep learning powered by advancements in hardware and data availability.
More Related Content
What's hot (20)
Ds for finance day 2
Ds for finance day 2QuantUniversity This document outlines the agenda and content for Day 2 of a 4-day course on Data Science for Finance. Day 2 focuses on building machine learning models, including supervised and unsupervised algorithms. The day consists of four modules: exploring data sets in Python, building a first model using credit risk data, evaluating models using different metrics, and deploying models for prediction. Key machine learning concepts that will be covered include linear regression, logistic regression, neural networks, K-means clustering, and anomaly detection. Labs allow attendees to apply these techniques to a credit risk case study.
Deep learning QuantUniversity meetup
Deep learning QuantUniversity meetupQuantUniversity This document summarizes a deep learning tutorial presentation given by Sri Krishnamurthy. The presentation introduced machine learning and deep learning concepts. It provided demonstrations of deep learning techniques like convolutional neural networks using Python libraries like Theano and Keras. Applications discussed included image recognition, speech recognition, question answering and cybersecurity. Recurrent neural networks and autoencoders were also introduced. The presenter emphasized the role of data, hardware and new approaches in advancing deep learning research.
Nlp and Neural Networks workshop
Nlp and Neural Networks workshopQuantUniversity The document is a comprehensive primer on neural network models specifically tailored for natural language processing (NLP), covering foundational concepts, applications, and various architectures such as recurrent and convolutional neural networks. It discusses the importance of understanding language, the challenges presented by ambiguity and context, and outlines applications in fields like finance and sentiment analysis. Additionally, it highlights training methods and the use of embeddings to improve word representation in NLP tasks.
An Introduction to Anomaly Detection
An Introduction to Anomaly DetectionKenneth Graham The document provides a comprehensive overview of anomaly detection, explaining its definition, types of anomalies, and various detection methods such as supervised, semi-supervised, and unsupervised techniques. It highlights the importance of understanding data context, types of input data, and relationships between instances for effective anomaly detection in applications like credit card fraud and medical data. Additionally, it discusses the use of neural networks, statistical methods, and graph-based techniques for identifying different kinds of anomalies.
Anomaly Detection for Real-World Systems
Anomaly Detection for Real-World SystemsManojit Nandi (1) Anomaly detection aims to identify data points that are noticeably different from expected patterns in a dataset. (2) Common approaches include statistical modeling, machine learning classification, and algorithms designed specifically for anomaly detection. (3) Streaming data poses unique challenges due to limited memory and need for rapid identification of anomalies. (4) Heuristics like z-scores and median absolute deviation provide robust ways to measure how extreme observations are compared to a distribution's center. (5) Density-based methods quantify how isolated data points are to identify anomalies. (6) Time series algorithms decompose trends and seasonality to identify global and local anomalous spikes and troughs.
Feature Selection for Document Ranking
Feature Selection for Document RankingAndrea Gigli The document discusses feature selection algorithms for learning to rank, focusing on proposed feature selection protocols and their application to publicly available web search data. It outlines various competing algorithms including naive greedy search and hierarchical clustering, detailing methods to measure feature relevance and selection performance. Additionally, it describes machine learning strategies for ranking and provides insights into the operational mechanics of ranking systems and their evaluation metrics.
Heuristic design of experiments w meta gradient search
Heuristic design of experiments w meta gradient searchGreg Makowski The document discusses heuristic design techniques for experimenting with model training parameters in data mining, aiming to help data miners navigate a plethora of algorithms and choices while maintaining organization and planning. It emphasizes the importance of structured methodologies, such as meta-gradient search, and the use of a model notebook to track and evaluate model performance effectively. Key concepts include regression testing, decision trees, and neural networks, along with strategies to minimize volatility and enhance reliability in model deployment.
Musings of kaggler
Musings of kagglerKai Xin Thia The document discusses the author's approach to a Kaggle competition by first preparing the data. Key steps included identifying outliers using median absolute deviation, using a Kolmogorov-Smirnov test to identify different distributions in the training data, and removing data that did not need to be modeled. The author then created various features from the data and experimented with different models before creating an ensemble of models. The ensemble approach combined predictions from random forests, gradient boosted models, support vector machines, and generalized linear models through a stacking regression. This process resulted in a top ranking of 49 out of 532 for the competition.
Biostatistics Workshop: Missing Data
Biostatistics Workshop: Missing DataHopkinsCFAR This document discusses approaches for handling missing data in statistical analyses. It begins by contrasting an ideal scenario with no missing data against the real-world problem of missing data. Common types of missing data are defined, including missing completely at random (MCAR), missing at random (MAR), and missing not at random (MNAR). Complete case analysis is described as biased except for large samples with little missing data. Alternative approaches like multiple imputation are presented along with their advantages and disadvantages. An example using indicator variables to handle missing clinic data in a regression is provided. Issues like creating indicators for each missing covariate are also noted.
Machine Learning Algorithm & Anomaly detection 2021
Machine Learning Algorithm & Anomaly detection 2021Chakrit Phain This document discusses various techniques for anomaly detection using machine learning, including:
1. It introduces machine learning and anomaly detection.
2. It covers different machine learning techniques for anomaly detection like genetic algorithms, Monte Carlo simulation, reinforcement learning, and generative adversarial networks.
3. It describes different types of anomalies like point, contextual, and collective anomalies. It also discusses techniques for identifying outliers.
4. Specific anomaly detection techniques are discussed, including recurrent neural networks, historical analysis with DBSCAN clustering, time shift detection, and text pattern anomaly detection using techniques like cosine similarity and autoencoders.
Missing Data and data imputation techniques
Missing Data and data imputation techniquesOmar F. Althuwaynee The document discusses techniques for imputing missing data (<NA>) in R. It introduces common imputation methods like MICE, missForest, and Hmisc. MICE creates multiple imputations using chained equations to account for uncertainty, while missForest uses random forests to impute missing values. Hmisc offers functions to impute missing values using methods like mean, regression, and predictive mean matching. The goal is to understand missing data, learn imputation methods, and choose the best approach for a given dataset.
Explainable AI Workshop
Explainable AI WorkshopQuantUniversity This document presents a workshop on a unified framework for model explanation, focusing on the concept of feature removal in Explainable AI (XAI). It outlines various methods and challenges in the field, emphasizing the need for a coherent theory to guide practitioners and researchers in effectively interpreting machine learning models. The talk concludes with the importance of counterfactual reasoning and the potential trade-offs in choosing appropriate methods for model explanation.
Machine Learning for Dummies
Machine Learning for DummiesVenkata Reddy Konasani This document provides an introduction to machine learning, including:
- It discusses how the human brain learns to classify images and how machine learning systems are programmed to perform similar tasks.
- It provides an example of image classification using machine learning and discusses how machines are trained on sample data and then used to classify new queries.
- It outlines some common applications of machine learning in areas like banking, biomedicine, and computer/internet applications. It also discusses popular machine learning algorithms like Bayes networks, artificial neural networks, PCA, SVM classification, and K-means clustering.
Imputation Techniques For Market Research Datasets With Missing Values
Imputation Techniques For Market Research Datasets With Missing Values Salford Systems This document discusses techniques for handling missing data in statistical analysis and modeling. It compares different modeling approaches on three datasets - one on shoe preferences from a stated preference survey, one on diabetes risk factors, and one on homeowner characteristics. It finds that classification and regression tree (CART) and multivariate adaptive regression splines (MARS) techniques are preferred for imputing missing values when the data contains mixed variable types and interactions among variables. CART can sequentially impute missing values for each variable while preserving the multivariate structure of the data.
Linear regression on 1 terabytes of data? Some crazy observations and actions
Linear regression on 1 terabytes of data? Some crazy observations and actionsHesen Peng 1) The document discusses using linear regression on 1 terabyte of data by leveraging Amazon Web Services' free tier and distributed computing algorithms in Python and R.
2) It notes the challenges of going beyond linear models with big data, including better prediction and real-time analytics.
3) A proposed solution is "universal association discovery" to find relationships between random variables regardless of form using functions on observation graphs, though this approach currently only works for continuous variables.
Aggregating Multiple Dimensions for Computing Document Relevance
Aggregating Multiple Dimensions for Computing Document RelevanceJosé Ramón Ríos Viqueira The document discusses a multi-criteria framework for evaluating document relevance, emphasizing the importance of various factors such as user profiles, document content, and specific criteria in retrieving relevant documents. It presents multiple case studies that demonstrate methods for combining and scoring documents based on different dimensions, including textual and metadata layers. The document ultimately aims to enhance information retrieval systems by adopting more nuanced and personalized approaches to scoring and relevance assessment.
udacity-dandsyllabus
udacity-dandsyllabusBora Yüret This document provides a syllabus for the Data Analyst Nanodegree program. It outlines 7 projects that students will complete to learn data analysis skills in Python, R, SQL, and MongoDB. The projects involve analyzing San Francisco bike share data, testing psychological effects, investigating datasets, wrangling OpenStreetMap data, exploring and summarizing data in R, introductory machine learning to detect fraud, and designing effective data visualizations. Supporting lessons teach statistical techniques, data wrangling, data visualization, and A/B testing.
Andrew Bossy. Data Imputation Using Reverse ML
Andrew Bossy. Data Imputation Using Reverse MLLviv Startup Club This document discusses various methods for data imputation and restoration using reverse machine learning. It describes types of missing data and common imputation techniques like mean, median, mode, regression, KNN, and MICE. Advanced generative models for imputation including VAE, GAIN, MisGAN, and VIGAN are also covered. The document proposes training autoencoders with dropout and regularization as a reverse machine learning technique for data restoration, finding it achieves better accuracy than classical methods.
Statistical Approaches to Missing Data
Statistical Approaches to Missing DataDataCards The document discusses the challenges of missing data in structured datasets, defining types of missingness (MCAR, MAR, and MNAR) and their implications for statistical analysis. It introduces various methods for handling missing data, such as imputation, interpolation, and the need for appropriate techniques depending on the nature of the missing data. The importance of understanding the mechanisms of missingness and their impact on data analysis outcomes is emphasized.
Missing Data and Causes
Missing Data and Causesakanni azeez olamide Missing data occurs when no data value is stored for a variable in an observation, usually due to manual errors or incorrect measurements. There are three types of missing data: missing completely at random, missing at random, and missing not at random. Several methods can be used to deal with missing data, including reducing the dataset, treating missing values as a special value, replacing with the mean, replacing with the most common value, and using the closest fit to impute missing values. Proper handling of missing data is important to avoid bias and distortions in analyzing the data.
Viewers also liked (20)
Deep learning Tutorial - Part II
Deep learning Tutorial - Part IIQuantUniversity The document is a presentation by Sri Krishnamurthy at a Quantuniversity meetup, covering various aspects of deep learning, specifically focusing on neural networks, autoencoders, and recurrent neural networks (RNNs). It outlines upcoming workshops, provides a recap of concepts like supervised and unsupervised learning, and discusses practical applications of neural networks in areas such as sentiment analysis and forecasting. The presentation emphasizes the significance of tools like Keras for implementing deep learning techniques.
Deep learning - Part I
Deep learning - Part IQuantUniversity The document presents an introduction to deep learning during a meetup hosted by Quantuniversity, highlighting the significance and applications of neural networks. Sri Krishnamurthy, the founder of Quantuniversity, discusses various tools and techniques in analytics, including Keras and Theano, and outlines future events related to deep learning education. It emphasizes the evolution and potential of deep learning powered by advancements in hardware and data availability.
Deep learning and Apache Spark
Deep learning and Apache SparkQuantUniversity The document is a presentation about a QuantUniversity meetup on deep learning and Apache Spark, featuring Sri Krishnamurthy, who discusses various deep learning frameworks and their integration with Spark for efficient data processing. It highlights the use of technologies like TensorFlow, BigDL, and DeepLearning4J for distributed deep learning, showcasing benefits such as reduced training time and resource efficiency. Upcoming workshops and events related to deep learning are also announced, along with resources for further learning.
Model Risk Management : Best Practices
Model Risk Management : Best PracticesQuantUniversity The document presents best practices for managing model risk in financial analytics, detailing concepts such as stress testing, scenario testing, and model validation. It outlines challenges in implementing model risk management and proposes solutions leveraging advanced technologies for analytics scalability, alongside methodologies for value-at-risk calculations. Additionally, it offers insights into model lifecycle management and invites participation in a beta program for its model risk analytics system.
Missing data handling
Missing data handlingQuantUniversity The document is a presentation on missing data analysis methods, presented by Sri Krishnamurthy at the Boston Data Festival in 2016. It discusses the challenges and various approaches for handling missing data, including deletion and imputation methods, along with their advantages and disadvantages. The presentation emphasizes the importance of minimizing bias and maximizing the use of available information in data analysis.
Energy analytics with Apache Spark workshop
Energy analytics with Apache Spark workshopQuantUniversity The document discusses energy analytics and the use of Apache Spark for large-scale data processing in the energy sector, focusing on customer segmentation and load forecasting use cases. It outlines the capabilities of Spark, highlighting its speed, ease of use, and integration with various data sources. The material also provides setup instructions for using Spark and discusses relevant algorithms such as k-means for clustering and regression for forecasting.
FitchLearning QuantUniversity Model Risk Presentation
FitchLearning QuantUniversity Model Risk PresentationQuantUniversity The document outlines a comprehensive framework for model risk management, emphasizing the importance of confidentiality, regulatory compliance, and the integration of model validation and verification processes. It discusses challenges financial institutions face regarding model risk, such as complexity and the need for customized approaches, along with best practices to enhance model risk management systems. The presentation also highlights technological advancements in analytics and machine learning that can aid in quantifying and managing model risk effectively.
PythonQuants conference - QuantUniversity presentation - Stress Testing in th...
PythonQuants conference - QuantUniversity presentation - Stress Testing in th...QuantUniversity The document discusses the scaling of Python models for model verification and stress testing in the financial sector, emphasizing the importance of model risk management and regulatory compliance. It highlights the need for robust stress testing frameworks, challenges in deploying and scaling models, and the growing adoption of Python among financial institutions due to its quant-friendly packages. Additionally, it presents case studies on value-at-risk and conditional value-at-risk methodologies implemented in Python, MATLAB, and R.
Stormwater analytics with MongoDB and Pentaho
Stormwater analytics with MongoDB and PentahoDave Callaghan Sparks Ignite, Inc. is a technology consulting firm focused on delivering outcomes through data integration, analytics, and management. They specialize in stormwater services aiming to improve water quality and reduce flood risks, employing the Internet of Things and real-time analytics for data-driven decision-making. The firm offers various consulting services across multiple sectors including financial services, government, and healthcare.
Model Risk Management: Using an infinitely scalable stress testing platform f...
Model Risk Management: Using an infinitely scalable stress testing platform f...QuantUniversity This document discusses model risk management, emphasizing a framework-driven approach to model verification and validation for financial institutions. It highlights the importance of addressing model risk due to its potential adverse consequences and outlines key processes such as model governance, lifecycle management, and stress testing. The presentation also covers advancements in technology that enhance model risk management and provides examples of model verification applications.
Guest talk- Roof Classification
Guest talk- Roof ClassificationQuantUniversity This document discusses classifying roof styles from aerial images using convolutional neural networks. It presents different approaches for roof style classification, including regular neural networks, regular networks with edge detection, and convolutional neural networks. It details the data used, packages implemented, network architectures, and reports the accuracy obtained from different models. The goal is to benchmark the accuracy of regular versus convolutional networks for roof style classification from satellite images.
Big data, Analytics and Beyond
Big data, Analytics and BeyondQuantUniversity The document discusses the evolution and impact of big data and analytics, emphasizing its importance for gaining a competitive advantage through technology. It highlights five key technological trends over the last decade, including the rise of machine learning and the shift to cloud computing. Companies are advised to assess their analytical capabilities and the business value of implementing big data solutions before proceeding.
A Framework Driven Approach to Model Risk Management (www.dataanalyticsfinanc...
A Framework Driven Approach to Model Risk Management (www.dataanalyticsfinanc...QuantUniversity The document presents a framework-driven approach to model risk management in financial institutions, highlighting the importance of effective model governance, lifecycle management, and validation processes. It discusses the challenges faced in quantifying and managing model risk due to complex quantitative systems and regulatory requirements. Additionally, it emphasizes leveraging technology and analytics for stress testing and scenario analysis to enhance model risk management practices.
Anomaly detection
Anomaly detectionQuantUniversity The document discusses anomaly detection techniques and best practices, focusing on graphical, statistical, and machine learning approaches. It highlights methodologies such as boxplots, chi-square tests, and clustering methods like k-means and local outlier factor. Additionally, it provides an overview of a full workshop on anomaly detection scheduled for July 2016 in Boston.
Top 10 Data Science Practitioner Pitfalls
Top 10 Data Science Practitioner PitfallsSri Ambati The document outlines the top 10 pitfalls in data science practice, including the importance of properly partitioning data into training, validation, and test sets, managing class imbalance, and addressing missing data and outliers. It emphasizes the significance of avoiding data leakage and creating actionable models, as well as the principle that there is no universally optimal algorithm or data preparation method. Practitioners are encouraged to understand their data and models deeply, iterating on approaches to enhance performance.
H2O World - NCS Continuous Media Optimization w/H2O - Satya Satyamoorthy
H2O World - NCS Continuous Media Optimization w/H2O - Satya SatyamoorthySri Ambati This document discusses Nielsen Catalina Solutions' (NCS) use of H2O for continuous media optimization. NCS matches consumer television viewing data with purchase data to target digital advertising. H2O allows NCS to build predictive models that estimate the sales lift from advertising exposures using a gradient boosting machine approach. NCS then uses these models to continuously optimize digital campaigns in real-time to maximize incremental sales.
Scaling spark
Scaling sparkAlex Rovner This document discusses how a media platform scaled their use of Spark across AWS to process terabytes of data daily. They moved from two on-premise clusters to running analytics and streaming workloads on AWS while keeping their core workload on-premise, initially using Spark on EMR but then self-managing Spark on EC2 for more flexibility. They implemented auto-scaling of the AWS clusters to maintain utilization targets and handle fluctuating workload demands.
Debugging & Tuning in Spark
Debugging & Tuning in SparkShiao-An Yuan This document provides tips and best practices for debugging and tuning Spark applications. It discusses Spark concepts like RDDs, transformations, actions, and the DAG execution model. It then gives recommendations for improving correctness, reducing overhead from parallelism, avoiding data skew, and tuning configurations like storage level, number of partitions, executor resources and joins. Common failures are analyzed along with their causes and fixes. Overall it emphasizes the importance of tuning partitioning, avoiding shuffles when possible, and using the right configurations to optimize Spark jobs.
Spark After Dark - LA Apache Spark Users Group - Feb 2015
Spark After Dark - LA Apache Spark Users Group - Feb 2015Chris Fregly The document presents insights into various technologies and paradigms related to Spark, including Spark Streaming, Machine Learning, Graph Processing, and the Lambda Architecture. It details the functionalities, use cases, and enhancements of Spark's capabilities and APIs, alongside methods for deploying Spark in cluster environments. Key topics include execution optimizations, fault tolerance, and the integration of the Spark ecosystem with other big data technologies.
Driving In-Store Sales with Real-Time Personalization - Cyril Nigg, Catalina ...
Driving In-Store Sales with Real-Time Personalization - Cyril Nigg, Catalina ...Sri Ambati Catalina is a data-driven company that enhances in-store and online sales through real-time personalized digital media, aiming to improve the effectiveness of promotional circulars and one-to-one shopper communication. With vast data capabilities, including billions of shopping trips and unique shopper IDs, Catalina's approach focuses on delivering relevant product recommendations and measurable incremental sales to retailers. The company's success is anchored on three key elements: a data management platform (DMP), aggregation API for real-time insights, and efficient monitoring for execution.
Ad
Similar to Scaling Analytics with Apache Spark (20)
Technical_Report_on_ML_Library
Technical_Report_on_ML_LibrarySaurabh Chauhan This technical report compares the machine learning libraries H2O and SparkMLlib. It finds that H2O offers more advanced machine learning algorithms and faster performance for tasks like linear modeling and random forests. Experiments show predictive analysis using H2O's GLM took 15.86 seconds and achieved 71.33% accuracy on iris data, while SparkMLlib's GLM took 25.70 seconds and had lower 68.33% accuracy. The report concludes H2O provides more efficient algorithms and integration with Spark for machine learning on big data.
Spark Summit EU talk by Josef Habdank
Spark Summit EU talk by Josef HabdankSpark Summit The presentation discusses an ensemble approach for large-scale data science, specifically focusing on predicting airfare prices using machine learning models trained with Spark and Python. It highlights the advantages of parallel online training, the importance of clustering methods, and the utilization of Gaussian mixture models for fuzzy clustering. The speaker emphasizes the efficacy of this methodology in processing billions of rows daily while maintaining accurate predictions and the infrastructure required for such implementations.
Scalable Monitoring Using Prometheus with Apache Spark Clusters with Diane F...
Scalable Monitoring Using Prometheus with Apache Spark Clusters with Diane F...Databricks The document discusses the integration of Prometheus for scalable monitoring of Apache Spark jobs, emphasizing the benefits of observability for performance tuning. It covers topics such as Spark features, JVM instrumentation, and how to monitor metrics using Prometheus and Grafana dashboards. Additionally, it explores memory optimization strategies in Spark and compares the performance of cached versus non-cached DataFrames.
Recent Developments In SparkR For Advanced Analytics
Recent Developments In SparkR For Advanced AnalyticsDatabricks The document discusses the SparkR package, which bridges the gap between R and big data analytics, providing descriptive and predictive analytics on large datasets. It outlines the architecture, features, and implementation of SparkR, including data frame operations, statistical computations, and integration with Spark's machine learning library. Future directions for SparkR include improving API consistency, enhancing R formula support, and integrating more machine learning algorithms.
Spark + AI Summit 2019: Headaches and Breakthroughs in Building Continuous Ap...
Spark + AI Summit 2019: Headaches and Breakthroughs in Building Continuous Ap...Landon Robinson The document discusses the complexities and benefits of Spark Streaming for building continuous applications, highlighting its powerful capabilities in processing live data streams. Key topics include testing, monitoring, batch interval optimization, helpful configurations, and transformations, along with actionable takeaways for more effective use of Spark Streaming. The authors, Landon Robinson and Jack Chapa, share their experiences and strategies for leveraging Spark in data-driven environments.
Headaches and Breakthroughs in Building Continuous Applications
Headaches and Breakthroughs in Building Continuous ApplicationsDatabricks The document discusses Spark Streaming and its complexities, highlighting its powerful capabilities for processing live data streams efficiently. It covers essential topics such as testing, monitoring, batch intervals, and resource management, providing practical takeaways for optimizing Spark Streaming applications. Additionally, it emphasizes the importance of configuration settings and techniques for managing offsets and transformations in streaming applications.
Spark ml streaming
Spark ml streamingAdam Doyle The document discusses Spark streaming and machine learning concepts like logistic regression, linear regression, and clustering algorithms. It provides code examples in Scala and Python showing how to perform binary classification on streaming data using Spark MLlib. Links and documentation are referenced for setting up streaming machine learning pipelines to train models on streaming data in real-time and make predictions.
Strata San Jose 2016: Scalable Ensemble Learning with H2O
Strata San Jose 2016: Scalable Ensemble Learning with H2OSri Ambati This document discusses scalable ensemble learning using the H2O platform. It provides an overview of ensemble methods like bagging, boosting, and stacking. The stacking or Super Learner algorithm trains a "metalearner" to optimally combine the predictions from multiple "base learners". The H2O platform and its Ensemble package implement Super Learner and other ensemble methods for tasks like regression and classification. An R code demo is presented on training ensembles with H2O.
Graph Analytics in Spark
Graph Analytics in SparkPaco Nathan Graph analytics can be used to analyze a social graph constructed from email messages on the Spark user mailing list. Key metrics like PageRank, in-degrees, and strongly connected components can be computed using the GraphX API in Spark. For example, PageRank was computed on the 4Q2014 email graph, identifying the top contributors to the mailing list.
Prediction as a service with ensemble model in SparkML and Python ScikitLearn
Prediction as a service with ensemble model in SparkML and Python ScikitLearnJosef A. Habdank The document discusses an ensemble modeling approach for large-scale data science, specifically for predicting airfare prices using SparkML and Python's Scikit-learn. It highlights the need for a robust architecture that supports online learning and parallel model training on billions of rows daily, showcasing techniques for clustering and optimizing model predictions. The author emphasizes the effectiveness of combining various machine learning models and infrastructure for accurate and scalable prediction services in the aviation industry.
Microservices, containers, and machine learning
Microservices, containers, and machine learningPaco Nathan This document discusses the principles and examples of using Apache Spark for graph analytics and machine learning, focusing on components like GraphX, optimization techniques, and the application of various algorithms. It highlights data preparation methods, data mining of email archives, and microservices in the context of processing and analyzing large datasets. Additionally, it provides code examples and resources for implementing graph analytics and text analysis with Spark.
04 open source_tools
04 open source_toolsMarco Quartulli This document discusses tools for distributed data analysis including Apache Spark. It is divided into three parts:
1) An introduction to cluster computing architectures like batch processing and stream processing.
2) The Python data analysis library stack including NumPy, Matplotlib, Scikit-image, Scikit-learn, Rasterio, Fiona, Pandas, and Jupyter.
3) The Apache Spark cluster computing framework and examples of its use including contexts, HDFS, telemetry, MLlib, streaming, and deployment on AWS.
SCALABLE MONITORING USING PROMETHEUS WITH APACHE SPARK
SCALABLE MONITORING USING PROMETHEUS WITH APACHE SPARKzmhassan The document presents a detailed overview of scalable monitoring using Prometheus with Apache Spark, highlighting the speakers' backgrounds and their work on performance analysis and data science applications. It covers key concepts such as observability, Spark and Prometheus functionalities, tuning Spark jobs, and using Grafana for visualizing performance data. The discussion includes practical examples and experiments that demonstrate improvements in memory usage and performance by comparing cached and non-cached dataframes.
Combining Machine Learning frameworks with Apache Spark
Combining Machine Learning frameworks with Apache SparkDataWorks Summit/Hadoop Summit This document discusses combining machine learning frameworks like TensorFlow with Apache Spark. It describes how Spark can be used to schedule and distribute machine learning tasks across a cluster in order to speed up model training. Specific examples are provided of using TensorFlow for neural network training on image data and distributing those computations using Spark. The document also outlines Apache Spark MLlib and its DataFrame-based APIs for building machine learning pipelines that can be trained and deployed at scale.
Training Neural Networks
Training Neural NetworksDatabricks The document outlines a training series on neural networks focused on concepts and practical applications using Keras. It covers tuning, optimization, and training algorithms, alongside challenges such as overfitting and underfitting, and discusses the architecture and advantages of convolutional neural networks (CNNs). The content is designed for individuals interested in understanding deep learning fundamentals and applying them effectively.
New Developments in H2O: April 2017 Edition
New Developments in H2O: April 2017 EditionSri Ambati The document presents an overview of H2O.ai, a scalable machine learning platform developed by a team of experts in statistics and engineering, based in Mountain View, California. It highlights new developments such as stacked ensembles, deep learning capabilities, integration with Spark, and automatic machine learning (AutoML) functionality. Additionally, it details the architecture and various algorithms available within the H2O platform, as well as the resources provided for users to engage with the software.
Big Data, Bigger Analytics
Big Data, Bigger AnalyticsItzhak Kameli This document discusses using Apache Spark for big data analytics at an insurance pricing and customer analytics company called Earnix. It summarizes Earnix's business problems modeling large customer behavior data, how Spark helps address performance issues with their existing 10GB datasets, and improvements made to Spark's MLlib machine learning library. These include adding statistical functionality like covariance estimation to logistic regression models and optimizing algorithms to run efficiently on Spark. Benchmark results show Spark providing scalability by reducing algorithm run times as more nodes are added.
GraphX: Graph analytics for insights about developer communities
GraphX: Graph analytics for insights about developer communitiesPaco Nathan The document provides an overview of Graph Analytics in Spark. It discusses Spark components and key distinctions from MapReduce. It also covers GraphX terminology and examples of composing node and edge RDDs into a graph. The document provides examples of simple traversals and routing problems on graphs. It discusses using GraphX for topic modeling with LDA and provides further reading resources on GraphX, algebraic graph theory, and graph analysis tools and frameworks.
2014-10-20 Large-Scale Machine Learning with Apache Spark at Internet of Thin...
2014-10-20 Large-Scale Machine Learning with Apache Spark at Internet of Thin...DB Tsai This document discusses machine learning techniques for large-scale datasets using Apache Spark. It provides an overview of Spark's machine learning library (MLlib), describing algorithms like logistic regression, linear regression, collaborative filtering, and clustering. It also compares Spark to traditional Hadoop MapReduce, highlighting how Spark leverages caching and iterative algorithms to enable faster machine learning model training.
Ad
More from QuantUniversity (20)
AI in Finance and Retirement Systems: Insights from the EBRI-Milken Institute...
AI in Finance and Retirement Systems: Insights from the EBRI-Milken Institute...QuantUniversity On March 11, I had the privilege of joining esteemed experts Bridget Bearden from the Employee Benefit Research Institute and Randy Blaha from Nationwide Financial at the EBRI- Milken Institute Retirement Symposium at The Reach, The John F. Kennedy Center for the Performing Arts in Washington, D.C. The event brought together thought leaders, researchers, and policymakers to discuss pressing challenges and opportunities shaping the future of retirement security.
During my session, I explored the latest advancements in artificial intelligence (AI) and large language models (LLMs) and their implications for financial services and retirement planning. AI is rapidly transforming decision-making processes, risk assessment, and personalized financial guidance, offering both exciting opportunities and critical considerations for the industry.
Key Takeaways:
✅ AI’s Evolving Role in Finance – From improving retirement portfolio management to enhancing customer engagement, AI is making financial services more accessible, efficient, and intelligent.
✅ The Rise of AI Agents – Intelligent agents are streamlining tasks like research synthesis, compliance monitoring, and personalized financial education, but organizations must ensure responsible deployment and ethical considerations.
✅ AI Readiness & Validation – While some use cases are advancing rapidly, many organizations are still assessing their AI maturity and readiness. Strategic deployment and validation mechanisms remain key to unlocking AI’s full potential.
✅ QuSkillbridge.ai & AI for Upskilling – AI-powered platforms like QuantUniversity 's QuSkillbridge.ai platform are accelerating learning and professional development by automating course creation, making knowledge transfer more scalable and impactful.
The financial industry stands at a pivotal moment, where AI’s power can redefine retirement planning, enhance decision-making, and empower financial professionals. However, as adoption grows, so does the need for strong governance, validation, and transparency to ensure AI-driven solutions are both reliable and equitable.
Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitig...
Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitig...QuantUniversity The document outlines a guest lecture series at the QU AI Fall School 2024, focused on the use and governance of generative AI in large U.S. financial institutions. It features speakers, including Jacob Kosoff from Bank of America and Sri Krishnamurthy from Quantuniversity, covering topics such as AI in investing, applications for financial professionals, and governance issues. Contact information for further details and participation is provided.
EU Artificial Intelligence Act 2024 passed !
EU Artificial Intelligence Act 2024 passed !QuantUniversity The European Parliament has adopted its position on the Artificial Intelligence Act, aiming to create a harmonized regulatory framework for AI systems in the EU. The regulation focuses on promoting trustworthy AI while ensuring health, safety, and fundamental rights protections, thereby facilitating free market movement of AI-based goods and services. It also reinforces the union's values and supports innovation, particularly for small and medium enterprises, amidst evolving AI technology and associated risks.
Managing-the-Risks-of-LLMs-in-FS-Industry-Roundtable-TruEra-QuantU.pdf
Managing-the-Risks-of-LLMs-in-FS-Industry-Roundtable-TruEra-QuantU.pdfQuantUniversity The roundtable discussed the adoption of large language models (LLMs) in financial services, focusing on use cases, emerging risks, and management strategies. Key topics included the potential for LLMs in internal knowledge retrieval and automation, existing risks related to accuracy and privacy, and the evolving nature of jobs within financial institutions. Participants expressed the need for end-to-end risk management approaches while navigating the challenges and opportunities presented by LLM adoption.
PYTHON AND DATA SCIENCE FOR INVESTMENT PROFESSIONALS
PYTHON AND DATA SCIENCE FOR INVESTMENT PROFESSIONALSQuantUniversity The document details an information session on the 'Python and Data Science for Investment Professionals' program offered by CFA Institute, focusing on key concepts needed to utilize Python for financial data analysis. It covers topics such as the growing importance of data science, machine learning applications in finance, and practical case studies on analyzing financial datasets. The program provides a structured online learning experience, inclusive of hands-on labs and case studies, and is designed for financial professionals seeking to enhance their data science skills.
Qu for India - QuantUniversity FundRaiser
Qu for India - QuantUniversity FundRaiserQuantUniversity The document provides an overview of machine learning and artificial intelligence applications in finance, emphasizing its role in transforming investment strategies and risk management. It discusses key trends, case studies, and the importance of data science in enhancing financial analytics. Additionally, the speaker, Sri Krishnamurthy, highlights educational initiatives to train professionals in these technologies at QuantUniversity.
Ml master class for CFA Dallas
Ml master class for CFA DallasQuantUniversity Sri Krishnamurthy presents on machine learning and AI in finance. He discusses how the 4th industrial revolution is being driven by emerging technologies like AI, robotics, and 5G. Machine learning and AI are revolutionizing the finance industry by enabling real-time analytics, predictive analytics, and automating tasks. Sri outlines the machine learning workflow and key areas where machine learning is being applied in finance like trading strategies, risk management, and fraud detection.
Algorithmic auditing 1.0
Algorithmic auditing 1.0QuantUniversity The document presents an overview of algorithmic auditing in the context of finance, emphasizing its importance for ensuring compliance, transparency, and fairness in machine learning applications. It discusses various frameworks and processes for conducting audits, the challenges faced in auditing AI algorithms, and the necessary expertise for auditors. Additionally, it highlights the role of regulatory bodies and the need for continuous skill development in the auditing community to keep pace with advancements in machine learning.
Towards Fairer Datasets: Filtering and Balancing the Distribution of the Peop...
Towards Fairer Datasets: Filtering and Balancing the Distribution of the Peop...QuantUniversity The document discusses the fairness and representation issues in computer vision datasets, particularly focusing on the ImageNet hierarchy and its implications on underrepresented groups. It identifies major problems such as biased image representation, offensive labels, and the impact of human labeling on model performance. The document emphasizes the need for better dataset practices and responsible algorithmic development to address these biases and enhance fairness in machine learning applications.
Machine Learning: Considerations for Fairly and Transparently Expanding Acces...
Machine Learning: Considerations for Fairly and Transparently Expanding Acces...QuantUniversity 1) The document summarizes a presentation on considerations for using machine learning to expand access to credit in a fair and transparent manner.
2) It discusses how machine learning can be used across various functions at Discover Financial Services like underwriting, customer servicing, and collections.
3) The presentation addresses challenges of interpreting complex machine learning models, ensuring fairness, and mitigating bias in models.
Seeing what a gan cannot generate: paper review
Seeing what a gan cannot generate: paper reviewQuantUniversity The paper 'Seeing What a GAN Cannot Generate' explores the issue of mode collapse in Generative Adversarial Networks (GANs), where the generator fails to capture certain classes from the target distribution. The authors propose a method to analyze and visualize the omissions made by GANs at both distribution and instance levels, utilizing generated image segmentation statistics and layer inversions. Although the study does not provide solutions to mitigate mode collapse, it highlights crucial questions regarding GAN training and the implications of architectural choices.
AI Explainability and Model Risk Management
AI Explainability and Model Risk ManagementQuantUniversity This document introduces a speaker series hosted by QuantUniversity on AI explainability and model risk management. The next speaker will be Dr. Anupam Datta, a professor at Carnegie Mellon University and co-founder of Truera, who is passionate about enabling responsible adoption of AI. He has led groundbreaking research in areas like AI explainability and governance. Instructions are provided for registering to participate in an associated lab using a provided code.
Algorithmic auditing 1.0
Algorithmic auditing 1.0QuantUniversity The document discusses algorithmic auditing within the context of AI and machine learning in finance, emphasizing the importance of validating algorithms, data, and processes for effective implementation. It outlines five key aspects to consider during auditing, including use cases, data quality, model evaluation, operational environment, and auditing processes. Sri Krishnamurthy, the presenter, highlights the relevance of algorithmic auditing in addressing biases, ensuring fairness, and enhancing interpretability in financial applications.
Machine Learning in Finance: 10 Things You Need to Know in 2021
Machine Learning in Finance: 10 Things You Need to Know in 2021QuantUniversity The document details the kickoff for the QU Winter School 2021 and outlines key trends in AI, machine learning, and fintech. It emphasizes the transformative impact of machine learning in finance and includes information on the speaker's background, Quantuniversity's achievements, and upcoming educational opportunities. Additionally, it touches on the importance of data science, regulatory considerations, and the evolving landscape of financial technologies.
Bayesian Portfolio Allocation
Bayesian Portfolio AllocationQuantUniversity Thomas Wiecki gave a presentation on Bayesian portfolio allocation. He discussed how Bayesian statistics allows flexible modeling that accounts for uncertainty, unlike traditional mean-variance optimization. His company PyMC Labs uses probabilistic programming to specify portfolio allocation models and infer posterior distributions over parameters. This allows generating predictions across many possible future scenarios and making decisions by optimizing over a loss function. The full Bayesian model can incorporate changes over time, pool information across strategies, and correlations to provide a robust allocation.
The API Jungle
The API JungleQuantUniversity The document provides an overview of the QU Speaker Series hosted by Sri Krishnamurthy from QuantUniversity, which focuses on data science, quantitative finance, and machine learning training. It includes registration information, insights into the developer community, and resources for utilizing various API platforms, including FactSet and Refinitiv. Additionally, it emphasizes the importance of architecture that enables speed of discovery and self-service in data processing.
Constructing Private Asset Benchmarks
Constructing Private Asset BenchmarksQuantUniversity Emilian Belev presented on constructing private asset benchmarks. He discussed challenges benchmarking private equity funds and explored alternatives including benchmarking private funds against each other or public assets. He proposed using statistical distributions of fund performance over time to derive multi-period covariances between funds and construct efficient portfolios. Valuation models can also price public and private assets using the same risk aversion to determine appropriate public benchmark margins. Forecasting metrics like public market equivalent addresses issues with private fund return smoothing and reporting periodicity. Liability-driven benchmarking assesses the probability of meeting future liquidity needs.
Machine Learning Interpretability
Machine Learning InterpretabilityQuantUniversity The document outlines a presentation by Dr. Agus Sudjianto on machine learning and model risk, focusing on self-explanatory models and their interpretability. It details his background in model risk management at Wells Fargo and previous roles at Lloyds Banking Group and Bank of America. The presentation emphasizes techniques like local linear models for interpretability and diagnostics in deep neural networks, alongside providing insights into model simplification methods.
Responsible AI in Action
Responsible AI in ActionQuantUniversity This document provides information about an online event hosted by QuantUniversity on responsible AI. It includes details such as the date, host, and instructions for registering for the accompanying lab session. QuantUniversity is described as a Boston-based training organization that has trained over 1000 students in quantitative methods, data science, and machine learning using programming languages such as MATLAB, Python, and R.
Qu speaker series 14: Synthetic Data Generation in Finance
Qu speaker series 14: Synthetic Data Generation in FinanceQuantUniversity The document discusses a speaker series on synthetic data generation in finance, hosted by Sri Krishnamurthy from Quantuniversity, featuring Stefan Jansen who explains generative adversarial networks (GANs) and their application in generating realistic financial time series data. It outlines the motivations for using synthetic data in finance, introduces the TimeGAN architecture, and presents methods for evaluating the quality of generated data. The conclusion indicates that TimeGAN can create synthetic data that mimics actual stock prices, suggesting potential for further development in the field.
Recently uploaded (20)
Pause Travail 22 Hostiou Girard 12 juin 2025.pdf
Pause Travail 22 Hostiou Girard 12 juin 2025.pdfInstitut de l'Elevage - Idele Diaporama de la Pause travail 22 sur les rizières au Ghana - 12 juin 2025 - Intervenants Nathalie Hostiou et Pierre Girard
apidays New York 2025 - Life is But a (Data) Stream by Sandon Jacobs (Confluent)
apidays New York 2025 - Life is But a (Data) Stream by Sandon Jacobs (Confluent)apidays Life is But a (Data) Stream: Building Quality Data Pipelines
Sandon Jacobs, Senior Developer Advocate at Confluent
apidays New York 2025
API Management for Surfing the Next Innovation Waves: GenAI and Open Banking
May 14 & 15, 2025
------
Check out our conferences at https://www.apidays.global/
Do you want to sponsor or talk at one of our conferences?
https://apidays.typeform.com/to/ILJeAaV8
Learn more on APIscene, the global media made by the community for the community:
https://www.apiscene.io
Explore the API ecosystem with the API Landscape:
https://apilandscape.apiscene.io/
Hypothesis Testing Training Material.pdf
Hypothesis Testing Training Material.pdfAbdirahmanAli51 This is a training manual about the basic of hypothesis using SPSS and STATA.
Complied by Abdirahman Sheikhdon.
[email protected]
apidays New York 2025 - Beyond Webhooks: The Future of Scalable API Event Del...
apidays New York 2025 - Beyond Webhooks: The Future of Scalable API Event Del...apidays Beyond Webhooks: The Future of Scalable API Event Delivery
Phil Leggetter, Head of Developer Experience at Hookdeck
apidays New York 2025
API Management for Surfing the Next Innovation Waves: GenAI and Open Banking
May 14 & 15, 2025
------
Check out our conferences at https://www.apidays.global/
Do you want to sponsor or talk at one of our conferences?
https://apidays.typeform.com/to/ILJeAaV8
Learn more on APIscene, the global media made by the community for the community:
https://www.apiscene.io
Explore the API ecosystem with the API Landscape:
https://apilandscape.apiscene.io/
apidays New York 2025 - Breaking Barriers: Lessons Learned from API Integrati...
apidays New York 2025 - Breaking Barriers: Lessons Learned from API Integrati...apidays Breaking Barriers: Lessons Learned from API Integration with Large Hotel Chains and the Role of Standardization
Constantine Nikolaou, Manager Business Solutions Architect at Booking.com
apidays New York 2025
API Management for Surfing the Next Innovation Waves: GenAI and Open Banking
May 14 & 15, 2025
------
Check out our conferences at https://www.apidays.global/
Do you want to sponsor or talk at one of our conferences?
https://apidays.typeform.com/to/ILJeAaV8
Learn more on APIscene, the global media made by the community for the community:
https://www.apiscene.io
Explore the API ecosystem with the API Landscape:
https://apilandscape.apiscene.io/
THE FRIEDMAN TEST ( Biostatics B. Pharm)
THE FRIEDMAN TEST ( Biostatics B. Pharm)JishuHaldar The Friedman Test is a valuable non-parametric alternative to the
Repeated Measures ANOVA, allowing for the comparison of three or
more related groups when data is ordinal or not normally distributed.
By ranking data instead of using raw values, the test overcomes the
limitations of parametric tests, making it ideal for small sample sizes and
real-world applications in medicine, psychology, pharmaceutical
sciences, and education. However, while it effectively detects differences
among groups, it does not indicate which specific groups differ, requiring
further post-hoc analysis.
apidays New York 2025 - The Challenge is Not the Pattern, But the Best Integr...
apidays New York 2025 - The Challenge is Not the Pattern, But the Best Integr...apidays The Challenge is Not the Pattern, But the Best Integration
Yisrael Gross, CEO at Ammune.ai
apidays New York 2025
API Management for Surfing the Next Innovation Waves: GenAI and Open Banking
May 14 & 15, 2025
------
Check out our conferences at https://www.apidays.global/
Do you want to sponsor or talk at one of our conferences?
https://apidays.typeform.com/to/ILJeAaV8
Learn more on APIscene, the global media made by the community for the community:
https://www.apiscene.io
Explore the API ecosystem with the API Landscape:
https://apilandscape.apiscene.io/
Managed Cloud services - Opsio Cloud Man
Managed Cloud services - Opsio Cloud ManOpsio Cloud At Opsio, we specialize in delivering advanced cloud services that enable businesses to scale, transform, and modernize with confidence. Our core offerings focus on cloud management, digital transformation, and cloud modernization — all designed to help organizations unlock the full potential of their technology infrastructure.We take a client-first approach, blending industry-leading hosted technologies with strategic expertise to create tailored, future-ready solutions. Leveraging AI, automation, and emerging technologies, our services simplify IT operations, enhance agility, and accelerate business outcomes. Whether you're migrating to the cloud or optimizing existing cloud environments, Opsio is your partner in achieving sustainable, measurable success.
apidays New York 2025 - Boost API Development Velocity with Practical AI Tool...
apidays New York 2025 - Boost API Development Velocity with Practical AI Tool...apidays Boost API Development Velocity with Practical AI Tooling
Sumit Amar, VP of Engineering at WEX
apidays New York 2025
API Management for Surfing the Next Innovation Waves: GenAI and Open Banking
May 14 & 15, 2025
------
Check out our conferences at https://www.apidays.global/
Do you want to sponsor or talk at one of our conferences?
https://apidays.typeform.com/to/ILJeAaV8
Learn more on APIscene, the global media made by the community for the community:
https://www.apiscene.io
Explore the API ecosystem with the API Landscape:
https://apilandscape.apiscene.io/
2.5-DESPATCH-ORDINARY MAILS.pptxlminub7b7t6f7h7t6f6g7g6fg
2.5-DESPATCH-ORDINARY MAILS.pptxlminub7b7t6f7h7t6f6g7g6fgmk1227103 8hunubybub6v6vubug7gyvtvtvyb7g7h7hih8h8hij8jihih
apidays Singapore 2025 - Enhancing Developer Productivity with UX (Government...
apidays Singapore 2025 - Enhancing Developer Productivity with UX (Government...apidays Enhancing Developer Productivity with UX
Petrine Tang, UX Designer at Government Technology Agency
Faith Ang, Product Manager at Government Technology Agency
apidays Singapore 2025
Where APIs Meet AI: Building Tomorrow's Intelligent Ecosystems
April 15 & 16, 2025
------
Check out our conferences at https://www.apidays.global/
Do you want to sponsor or talk at one of our conferences?
https://apidays.typeform.com/to/ILJeAaV8
Learn more on APIscene, the global media made by the community for the community:
https://www.apiscene.io
Explore the API ecosystem with the API Landscape:
https://apilandscape.apiscene.io/
apidays Singapore 2025 - 4 Identity Essentials for Scaling SaaS in Large Orgs...
apidays Singapore 2025 - 4 Identity Essentials for Scaling SaaS in Large Orgs...apidays 4 identity factors you didn't know you needed to support large organizations in your SaaS
Daizen Ikehara, Principal Developer Advocate at Auth0
apidays Singapore 2025
Where APIs Meet AI: Building Tomorrow's Intelligent Ecosystems
April 15 & 16, 2025
------
Check out our conferences at https://www.apidays.global/
Do you want to sponsor or talk at one of our conferences?
https://apidays.typeform.com/to/ILJeAaV8
Learn more on APIscene, the global media made by the community for the community:
https://www.apiscene.io
Explore the API ecosystem with the API Landscape:
https://apilandscape.apiscene.io/
apidays Singapore 2025 - Building Finance Innovation Ecosystems by Umang Moon...
apidays Singapore 2025 - Building Finance Innovation Ecosystems by Umang Moon...apidays Building Finance Innovation Ecosystems
Umang Moondra, CEO at APIX
apidays Singapore 2025
Where APIs Meet AI: Building Tomorrow's Intelligent Ecosystems
April 15 & 16, 2025
------
Check out our conferences at https://www.apidays.global/
Do you want to sponsor or talk at one of our conferences?
https://apidays.typeform.com/to/ILJeAaV8
Learn more on APIscene, the global media made by the community for the community:
https://www.apiscene.io
Explore the API ecosystem with the API Landscape:
https://apilandscape.apiscene.io/
Media_Literacy_Index_of_Media_Sector_Employees.pdf
Media_Literacy_Index_of_Media_Sector_Employees.pdfOlhaTatokhina1 Report based on the findings of a quantitative research conducted by the research agency New Image Marketing Group, commissioned by the NGO Detector Media, compiled by PhD in Sociology Marta Naumova.
KLIP2Data voor de herinrichting van R4 West en Oost
KLIP2Data voor de herinrichting van R4 West en Oostjacoba18 Ontwikkeling van een centrale GIS-databank voor de periodieke import van KLIP-data, BIM- en CAD-modellen van het R4-project en bijhorende field app om de databank zowel online als offline te raadplegen en bij te werken.
QUALITATIVE EXPLANATORY VARIABLES REGRESSION MODELS
QUALITATIVE EXPLANATORY VARIABLES REGRESSION MODELSAmeya Patekar QUALITATIVE EXPLANATORY VARIABLES REGRESSION MODELS
SQL-Demystified-A-Beginners-Guide-to-Database-Mastery.pptx
SQL-Demystified-A-Beginners-Guide-to-Database-Mastery.pptxbhavaniteacher99 Unlock the power of data with “SQL Demystified” — your perfect beginner-friendly guide to mastering Structured Query Language (SQL)! Whether you're a student, aspiring analyst, or a professional stepping into the world of data, this detailed and practical PowerPoint presentation simplifies the fundamentals and builds your SQL skills step-by-step.
📘 What’s Inside?
Introduction to SQL and its real-world importance
Understanding database structures: tables, columns & relationships
Core SQL queries: SELECT, FROM, WHERE, ORDER BY
Hands-on examples and practice exercises
Advanced SQL: JOIN, subqueries, and aggregate functions
Data manipulation: INSERT, UPDATE, DELETE
Performance tips and indexing basics
SQL security best practices and learning paths for future growth
🧠 Who is it for?
Beginners eager to learn SQL
Data science and computer students
Business professionals and analysts
Anyone curious about how databases work
✅ Features:
Clear visuals and examples
Easy-to-follow explanations
Real-life applications in banking, healthcare, e-commerce, and more
Reusable and editable presentation file (PPTX format)
Take your first step toward data mastery with confidence. Learn SQL the smart way — with structure, clarity, and purpose.
Scaling Analytics with Apache Spark
- 1. Location: QuantUniversity Meetup August 8th 2016 Boston MA Scaling Analytics with Apache Spark 2016 Copyright QuantUniversity LLC. Presented By: Sri Krishnamurthy, CFA, CAP www.QuantUniversity.com [email protected]
- 2. 2 Slides and Code will be available at: http://www.analyticscertificate.com/SparkWorkshop/
- 3. - Analytics Advisory services - Custom training programs - Architecture assessments, advice and audits
- 4. • Founder of QuantUniversity LLC. and www.analyticscertificate.com • Advisory and Consultancy for Financial Analytics • Prior Experience at MathWorks, Citigroup and Endeca and 25+ financial services and energy customers (Shell, Firstfuel Software etc.) • Regular Columnist for the Wilmott Magazine • Author of forthcoming book “Financial Modeling: A case study approach” published by Wiley • Charted Financial Analyst and Certified Analytics Professional • Teaches Analytics in the Babson College MBA program and at Northeastern University, Boston Sri Krishnamurthy Founder and CEO 4
- 5. 5 Quantitative Analytics and Big Data Analytics Onboarding • Trained more than 500 students in Quantitative methods, Data Science and Big Data Technologies using MATLAB, Python and R • Launching the Analytics Certificate Program in September
- 6. (MATLAB version also available)
- 7. 7 Quantitative Analytics and Big Data Analytics Onboarding • Apply at: www.analyticscertificate.com • Program starting September 18th • Module 1: ▫ Sep 18th , 25th , Oct 2nd, 9th • Module 2: ▫ Oct 16th , 23th , 30th, Nov 6th • Module 3: ▫ Nov 13th, 20th, Dec 4th, Dec 11th • Capstone + Certification Ceremony ▫ Dec 18th
- 8. 8 • August ▫ 14-20th : ARPM in New York www.arpm.co QuantUniversity presenting on Model Risk on August 14th ▫ 18-21st : Big-data Bootcamp http://globalbigdataconference.com/68/boston/big-data- bootcamp/event.html • September ▫ 1st : QuantUniversity Meetup (AnalyticsCertificate program open house) ▫ 11th, 12th : Spark Workshop, Boston ▫ 19th, 20th : Anomaly Detection Workshop, New York Events of Interest
- 9. 9
- 10. Agenda 1. A quick introduction to Apache Spark 2. A sample Spark Program 3. Clustering using Apache Spark 4. Regression using Apache Spark 5. Simulation using Apache Spark
- 11. Apache Spark : Soaring in Popularity Ref: Wall street Journal http://www.wsj.com/articles/newer-software-aims-to-crunch-hadoops-numbers-1434326008
- 12. What is Spark ? • Apache Spark™ is a fast and general engine for large-scale data processing. • Came out of U.C. Berkeley’s AMP Lab Lightning-fast cluster computing
- 13. Why Spark ? Speed Run programs up to 100x faster than Hadoop MapReduce in memory, or 10x faster on disk. Spark has an advanced DAG execution engine that supports cyclic data flow and in-memory computing.
- 14. Why Spark ? • text_file = spark.textFile("hdfs://...") text_file.flatMap(lambda line: l ine.split()) .map(lambda word: (word, 1)) .reduceByKey(lambda a, b: a+b) • Word count in Spark's Python API Ease of Use • Write applications quickly in Java, Scala or Python,R. • Spark offers over 80 high-level operators that make it easy to build parallel apps. And you can use it interactively from the Scala and Python shells. • R support recently added
- 15. Why Spark ? • Generality • Combine SQL, streaming, and complex analytics. • Spark powers a stack of high-level tools including: 1. Spark Streaming: processing real-time data streams 2. Spark SQL and DataFrames: support for structured data and relational queries 3. MLlib: built-in machine learning library 4. GraphX: Spark’s new API for graph processing
- 16. Why Spark? • Runs Everywhere • Spark runs on Hadoop, Mesos, standalone, or in the cloud. It can access diverse data sources including HDFS, Cassandra, HBase, and S3. • You can run Spark using its standalone cluster mode, on EC2, on Hadoop YARN, or on Apache Mesos. • Access data in HDFS, Cassandra, HBase, Hive, Tachyon, and any Hadoop data source.
- 17. Key Features of Spark • Handles batch, interactive, and real-time within a single framework • Native integration with Java, Python, Scala, R • Programming at a higher level of abstraction • More general: map/reduce is just one set of supported constructs
- 18. Secret Sauce : RDD, Transformation, Action
- 19. How does it work? • Resilient Distributed Datasets (RDD) are the primary abstraction in Spark – a fault-tolerant collection of elements that can be operated on in parallel. • Transformations create a new dataset from an existing one. All transformations in Spark are lazy: they do not compute their results right away – instead they remember the transformations applied to some base dataset. • Actions return a value to the driver program after running a computation on the dataset.
- 20. How is Spark different? • Map – Reduce : Hadoop
- 21. Problems with this MR model • Difficult to code
- 22. Getting started • http://spark.apache.org/docs/latest/index.html • http://datascience.ibm.com/ • https://community.cloud.databricks.com
- 24. Machine learning with Spark
- 25. Machine learning with Spark
- 26. 26 Machine learning with Spark
- 27. Use case 1 : Segmenting stocks • If we have a basket of stocks and their price history, how do we segment them into different clusters? • What metrics could we use to measure similarity? • Can we evaluate the effect of changing the number of clusters ? • Do the results seem actionable?
- 28. K-means Given a set of observations (x1, x2, …, xn), where each observation is a d-dimensional real vector, k-means clustering aims to partition the n observations into k (≤ n) sets S = {S1, S2, …, Sk} so as to minimize the within-cluster sum of squares (WCSS). In other words, its objective is to find: where μi is the mean of points in Si. http://shabal.in/visuals/kmeans/2.html
- 29. Demo • Kmeans spark case.ipynb
- 30. Use-case 2 – Regression • Given historical weekly interest data of AAA bond yields, 10 year treasuries, 30 year treasuries and Federal fund rates, build a regression model that fits • Changes to AAA = function of (Changes to 10year rates, Changes to 30 year rates, Changes to FF rates)
- 31. Linear regression • Linear regression investigates the linear relationships between variables and predict one variable based on one or more other variables and it can be formulated as: 𝑌 = 𝛽0 + 𝑖=1 𝑝 𝛽𝑖 𝑋𝑖 where Y and 𝑋𝑖 are random variables, 𝛽𝑖 is regression coefficient and 𝛽0 is a constant. • In this model, ordinary least squares estimator is usually used to minimize the difference between the dependent variable and independent variables. 31
- 32. Ordinary Least Squares Regression
- 35. Example: • Portfolio Growth • Given: ▫ INVESTMENT_INIT = 100000 # starting amount ▫ INVESTMENT_ANN = 10000 # yearly new investment ▫ TERM = 30 # number of years ▫ MKT_AVG_RETURN = 0.11 # percentage ▫ MKT_STD_DEV = 0.18 # standard deviation ▫ Run 10000 monte-carlo simulation paths and compute the expected value of the portfolio at the end of 30 years Ref: https://cloud.google.com/solutions/monte-carlo-methods-with-hadoop-spark
- 36. 36 • The count-distinct problem is the problem of finding the number of distinct elements in a data stream with repeated elements. • HyperLogLog is an algorithm for the count-distinct problem, approximating the number of distinct elements in a multiset • Calculating the exact cardinality of a multiset requires an amount of memory proportional to the cardinality, which is impractical for very large data sets. Probabilistic cardinality estimators, such as the HyperLogLog algorithm, use significantly less memory than this, at the cost of obtaining only an approximation of the cardinality. Hyperloglog Ref: https://en.wikipedia.org/wiki/HyperLogLog
- 37. 37 Hyperloglog The basis of the HyperLogLog algorithm is the observation that the cardinality of a multiset of uniformly distributed random numbers can be estimated by calculating the maximum number of leading zeros in the binary representation of each number in the set. If the maximum number of leading zeros observed is n, an estimate for the number of distinct elements in the set is 2^n Ref: https://en.wikipedia.org/wiki/HyperLogLog
- 38. 38 • Approximate algorithms ▫ approxCountDistinct: returns an estimate of the number of distinct elements ▫ approxQuantile: returns approximate percentiles of numerical data Refer: https://databricks-prod- cloudfront.cloud.databricks.com/public/4027ec902e239c93eaaa8714f17 3bcfc/8599738367597028/4196864626084292/3601578643761083/late st.html Demo from Databricks’s blog
- 39. 39 • As per Databricks’s blog: “Spark strives at implementing approximate algorithms that are deterministic (they do not depend on random numbers to work) and that have proven theoretical error bounds: for each algorithm, the user can specify a target error bound, and the result is guaranteed to be within this bound, either exactly (deterministic error bounds) or with very high confidence (probabilistic error bounds)” Spark’s implementation
- 42. 42 Q&A Slides, code and details about the Apache Spark Workshop at: http://www.analyticscertificate.com/SparkWorkshop/
- 43. Thank you! Members & Sponsors! Sri Krishnamurthy, CFA, CAP Founder and CEO QuantUniversity LLC. srikrishnamurthy www.QuantUniversity.com Contact Information, data and drawings embodied in this presentation are strictly a property of QuantUniversity LLC. and shall not be distributed or used in any other publication without the prior written consent of QuantUniversity LLC. 43