Data Science at Scale: Using Apache Spark for Data Science at Bitly
Download as pptx, pdf5 likes5,418 views
The document is a presentation by Sarah Guido on using Apache Spark for data science at Bitly, focusing on big data analysis, workflow setup, and live demonstrations of Spark's capabilities. It highlights the advantages of Spark over Hadoop, including speed and functionality, and discusses data processing, exploratory data analysis, and topic modeling. The talk concludes with current and future projects involving Spark, emphasizing its role in research and development.
1 of 38
Downloaded 20 times
Ad
Recommended
Spark: The Good, the Bad, and the Ugly
Spark: The Good, the Bad, and the UglySarah Guido Sarah Guido's talk at Data Day Texas 2016 shares her experiences and insights using Apache Spark, highlighting its advantages such as speed and ease of prototyping, as well as challenges like frequent updates and documentation issues. She discusses the importance of adopting Spark for big data analysis and the ongoing struggle between using Python and Scala due to discrepancies in features. Ultimately, Guido expresses hope for the future growth of the Spark community and improved resources for users.
Data Day Seattle 2015: Sarah Guido
Data Day Seattle 2015: Sarah GuidoBitly The document outlines Sarah Guido's presentation on utilizing Apache Spark for data science at Bitly, sharing her workflow and the tools within Spark. It emphasizes the need for efficient big data analysis and explores challenges faced in data processing, including exploratory data analysis and topic modeling. The talk also touches on future applications of Spark in production and ongoing projects at Bitly.
Analyzing Data With Python
Analyzing Data With PythonSarah Guido Sarah Guido gave a presentation on analyzing data with Python. She discussed several Python tools for preprocessing, analysis, and visualization including Pandas for data wrangling, scikit-learn for machine learning, NLTK for natural language processing, MRjob for processing large datasets in parallel, and ggplot for visualization. For each tool, she provided examples and use cases. She emphasized that the best tools depend on the type of data and analysis needs.
Apache Spark for Everyone - Women Who Code Workshop
Apache Spark for Everyone - Women Who Code WorkshopAmanda Casari This document outlines a workshop on Apache Spark, detailing its features, such as fast cluster computing, in-memory processing, and support for multiple programming languages. It emphasizes the importance of Spark for data science and machine learning applications, describing its capabilities for data querying and real-time processing. Additionally, it includes practical information on using Spark with various programming environments, such as Python and R, and offers resources for further learning.
State of Play. Data Science on Hadoop in 2015 by SEAN OWEN at Big Data Spain ...
State of Play. Data Science on Hadoop in 2015 by SEAN OWEN at Big Data Spain ...Big Data Spain The document discusses the current state of data science on Hadoop as presented by Sean Owen from Cloudera. It highlights the benefits of Hadoop, such as cost savings and the ability to handle larger data sets, while outlining a phased approach to implementing data science in operations. Additionally, it emphasizes the transition from exploratory analytics to operational analytics using frameworks like Lambda architecture for real-time machine learning.
PyCon Singapore 2013 Keynote
PyCon Singapore 2013 KeynoteWes McKinney This document summarizes a talk given by Wes McKinney on the future of Python and data analysis. It discusses how pandas has helped make Python a good language for data preparation and analysis, as well as trends like the rise of web/cloud computing and big data that Python needs to keep up with. It suggests embracing JavaScript more to make Python more web-friendly, and notes opportunities around data on the web, golden ages of web visualization, and leveraging new JIT compiler technologies to keep Python relevant for data.
Cascalog at May Bay Area Hadoop User Group
Cascalog at May Bay Area Hadoop User Groupnathanmarz Cascalog is a Clojure-based query language for Hadoop that provides a powerful and easy-to-use tool for data analysis. It allows users to write queries as regular Clojure code, offering features like joins, aggregators, functions, and sorting. Cascalog is unique in that it offers the full power of Clojure at all times by integrating queries directly into the programming language. BackType uses Cascalog for tasks like identifying influencers on social media, determining exposure to URLs, and studying engagement over time.
Semantic Search: Fast Results from Large, Non-Native Language Corpora with Ro...
Semantic Search: Fast Results from Large, Non-Native Language Corpora with Ro...Databricks The document discusses the use of Spark for enabling semantic search in foreign language corpora, addressing challenges such as machine translation difficulties and the need for effective document tagging and ranking. It highlights how Spark's distributed computing capabilities facilitate the processing of vast amounts of unstructured data to return relevant search results swiftly. Performance results indicate that the implementation allows for quick scraping and scoring of a large corpus, enhancing the speed and efficacy of the semantic search process.
Watching Pigs Fly with the Netflix Hadoop Toolkit (Hadoop Summit 2013)
Watching Pigs Fly with the Netflix Hadoop Toolkit (Hadoop Summit 2013)Jeff Magnusson The document discusses Netflix's Hadoop toolkit presented at the Hadoop Summit 2013, focusing on making data accessible and processable using various tools like Franklin, Sting, and Lipstick. It highlights the importance of data visualization and user empowerment in creating effective recommendations based on user preferences and content similarity. Additionally, it invites feedback and contributions to their open-source projects and provides links to related resources.
An excursion into Graph Analytics with Apache Spark GraphX
An excursion into Graph Analytics with Apache Spark GraphXKrishna Sankar The document discusses the use of Apache Spark's GraphX for graph processing, outlining its API landscape, algorithms, and real-world applications. It includes a tutorial agenda covering topics such as graph structures, community analysis, and case studies like the AlphaGo tweet analysis. Additionally, it highlights various graph processing frameworks and partitioning strategies to efficiently manage large datasets.
Solr for Data Science
Solr for Data ScienceGrant Ingersoll The document discusses Solr, an open-source search platform with over 8 million downloads, highlighting its key features such as full-text search, data analysis tools, and massive scalability. It emphasizes the importance of quickly identifying significant data for exploration and includes information on integrations with tools like Logstash and Kibana. Lucidworks, the company behind Solr, is noted for its substantial contributions to Solr's development, providing various resources and integration options for data science applications.
Pandas/Data Analysis at Baypiggies
Pandas/Data Analysis at BaypiggiesAndy Hayden The document discusses lessons learned in the performance and design of the Python library Pandas, highlighting its benefits for data analysis and the importance of optimizing user experience. Key topics include user productivity, workflow improvements, API design considerations, and performance tracking using tools like vbench. The authors emphasize the need for robust documentation and careful optimization to ensure a seamless and efficient user experience.
Scalable Monitoring Using Apache Spark and Friends with Utkarsh Bhatnagar
Scalable Monitoring Using Apache Spark and Friends with Utkarsh BhatnagarDatabricks The document discusses the challenges and requirements for scalable monitoring using Apache Spark and Grafana, highlighting the need for handling millions of unique metrics and billions of data points daily. It details specific requirements for various teams, addressing the architectural considerations for scaling infrastructure and supporting real-time monitoring. Additionally, it mentions past achievements, the capability to process large volumes of metrics and logs, and invites for collaboration and hiring.
Jake Mannix, Lead Data Engineer, Lucidworks at MLconf SEA - 5/20/16
Jake Mannix, Lead Data Engineer, Lucidworks at MLconf SEA - 5/20/16MLconf The document outlines a practical data science workbench using Spark and Solr, focusing on data engineering and machine learning applications. It discusses key features and workflow examples, including data exploration, classification, and building recommender systems. The integration of Spark with Solr enhances capabilities for processing large datasets efficiently while providing powerful analytical tools.
sparklyr - Jeff Allen
sparklyr - Jeff AllenSri Ambati The document discusses sparklyr, an open-source R package that facilitates the integration of the R programming language with Apache Spark for big data analysis. It highlights the features of sparklyr, including full dplyr support and integration with RStudio, as well as its capabilities in managing bigger-than-memory data. Additionally, it compares sparklyr with the existing sparkr and introduces the rsparkling extension for H2O integration.
Taking Jupyter Notebooks and Apache Spark to the Next Level PixieDust with Da...
Taking Jupyter Notebooks and Apache Spark to the Next Level PixieDust with Da...Databricks The document discusses the integration of developers into data science processes using tools like Jupyter Notebooks and Pixiedust for building data-driven applications, notably for performing real-time sentiment analysis with Apache Spark. It highlights the collaboration between software developers and data scientists to address data challenges, emphasizing that future solutions depend on both roles working together. Key features of Pixiedust, including visualizations and user-friendly interfaces for non-technical users, are presented as essential elements for democratizing data analysis.
Hadoop for Data Science
Hadoop for Data ScienceDonald Miner The document discusses the application of Hadoop in data science, highlighting its distributed file system (HDFS) and MapReduce framework for efficient data analysis. It features examples and cool aspects of Hadoop including linear scalability, schema on read, and its capability to handle unstructured data. Various data science tasks such as data exploration, classification, natural language processing, and recommender systems are covered, along with the importance of using appropriate tools for different phases of model training workflows.
Deduplication and Author-Disambiguation of Streaming Records via Supervised M...
Deduplication and Author-Disambiguation of Streaming Records via Supervised M...Spark Summit The document discusses advancements in deduplication and author disambiguation of scholarly records using supervised models and content encoders within the Elsevier platform, emphasizing the importance of efficient data processing techniques. It outlines the challenges and solutions for managing peer-reviewed publications in Scopus, focusing on machine learning and cloud scalability for real-time document analysis. The implementation of algorithms like word2vec for document encoding aims to improve recommendation systems and personalized user experiences across various academic entities.
From R Script to Production Using rsparkling with Navdeep Gill
From R Script to Production Using rsparkling with Navdeep GillDatabricks The document presents an overview of H2O.ai, a company specializing in open-source software for machine learning and data science, including its key components such as H2O Sparkling Water, Sparklyr, and Rsparkling. It explains the integration of H2O within the Spark ecosystem, demonstrating how users can leverage advanced machine learning algorithms in their Spark workflows using these packages. The document also includes an example use case for R and Spark integration, providing a practical demonstration of the technologies discussed.
Fulfilling Apache Arrow's Promises: Pandas on JVM memory without a copy
Fulfilling Apache Arrow's Promises: Pandas on JVM memory without a copyUwe Korn This document discusses how Apache Arrow enables sharing data between Python and Java without copying. It summarizes Arrow's capabilities for efficient in-memory columnar data and its ability to exchange data between different programming languages. The document then outlines how Arrow, through its Java and Python libraries, allows querying data in Java from Python without copying, by passing memory addresses between the two environments. This enables faster data science workflows that involve both Python and Java/Scala.
Indexing big data in the cloud
Indexing big data in the cloudOpenSource Connections The document discusses strategies for indexing big data in the cloud, highlighting the importance of agility, minimalism, and effective use of tools like Solr. It outlines a real project approach, emphasizing the need for quick prototypes and thorough testing while managing large datasets. Key technical details on server setup and scaling methods are also included, along with the potential alternatives for managing cloud resources.
The Little Warehouse That Couldn't Or: How We Learned to Stop Worrying and Mo...
The Little Warehouse That Couldn't Or: How We Learned to Stop Worrying and Mo...Spark Summit The document discusses the challenges faced by Shopify in using its existing data warehouse and ETL processes due to increasing data volume and complexity. It describes Shopify's attempts to use Pig and Luigi as well as Platfora to address these issues, but notes they did not meet Shopify's needs. Shopify then moved to using Spark due to its fast performance, nice development model using Python, and ability to better handle their data and query complexity. The summary provides an overview of why Shopify changed its data warehousing approach and the key technology it adopted.
Large Scale Image Forensics using Tika and Tensorflow [ICMR MFSec 2017]
Large Scale Image Forensics using Tika and Tensorflow [ICMR MFSec 2017]Thamme Gowda The document presents a methodology for large-scale image forensics using deep learning, focused on monitoring illegal weapons sales through the DARPA Memex program. It describes the integration of TensorFlow with Apache Tika for analyzing 1.4 million weapon images using the Inception-v3 model, optimized for efficient performance. The evaluation indicates successful image recognition, aiding the classification of weapons and enhancing data retrieval capabilities.
Rental Cars and Industrialized Learning to Rank with Sean Downes
Rental Cars and Industrialized Learning to Rank with Sean DownesDatabricks The document discusses the challenges and principles of industrializing data science workflows in cloud infrastructure, highlighting the need for clarity around data ownership and organization. It presents idiosyncratic operating principles that emphasize the distinct roles of engineers, product managers, and data scientists. The presentation also stresses the importance of fostering a data-centric culture and closing the loop on data utilization.
Congressional PageRank: Graph Analytics of US Congress With Neo4j
Congressional PageRank: Graph Analytics of US Congress With Neo4jWilliam Lyon The document discusses modeling the US Congress as a graph database using Neo4j and analyzing relationships between legislators to identify influential members. It describes loading congressional data into Neo4j, querying relationships between legislators and states they represent. Methods for identifying influential legislators include degree centrality, betweenness centrality, and PageRank computed on a bill co-sponsorship graph using both Neo4j and Apache Spark with GraphX.
Extracting Insights from Data at Twitter
Extracting Insights from Data at TwitterPrasad Wagle Prasad Wagle's talk discussed how Twitter extracts insights from its large volumes of data. Twitter collects hundreds of millions of tweets and interactions per day from over 300 million monthly active users, creating big data challenges around velocity, volume, and variety. Twitter stores this data in hundreds of petabytes across large Hadoop clusters and processes it using batch tools like Hadoop and Spark as well as real-time tools like Heron. Insights are generated through basic analytics like user counts, A/B testing of new features, and custom data science work including machine learning models for recommendations, content filtering, and ad targeting. Systems, programming, and statistical skills are needed to effectively extract value from Twitter's big data.
PyCon.DE / PyData Karlsruhe keynote: "Looking backward, looking forward"
PyCon.DE / PyData Karlsruhe keynote: "Looking backward, looking forward"Wes McKinney This document discusses the history and development of Python data analysis tools, including pandas. It covers Wes McKinney's work on pandas from 2008 to the present, including the motivations for making data analysis easier and more productive. It also summarizes the development of related projects like Apache Arrow for standardizing columnar data representations to improve code reuse across languages.
Converging Big Data and Application Infrastructure by Steven Poutsy
Converging Big Data and Application Infrastructure by Steven PoutsyBig Data Spain The document introduces the evolution of application infrastructure, highlighting advancements such as distributed software, easier programming languages, and improved statistical libraries. It emphasizes the integration of big data, APIs, and machine learning with web applications using tools like Docker, Kubernetes, and OpenShift. The document provides resources for setting up OpenShift and related tools for analysis and encourages contributions through pull requests.
Spark meetup v2.0.5
Spark meetup v2.0.5Yan Zhou The document discusses Huawei Technologies' implementation of Spark SQL on HBase, highlighting its motivations, performance capabilities, and future roadmap. It details the unique data challenges in telecommunications and how the technology can handle complex query types efficiently. Additionally, the paper reviews performance metrics, optimizations, and contributions to the Spark ecosystem from Huawei's big data teams.
DataEngConf SF16 - Spark SQL Workshop
DataEngConf SF16 - Spark SQL WorkshopHakka Labs The document outlines an agenda for an Apache Spark workshop, including topics such as Spark SQL, RDD operations, and user-defined functions (UDFs). It discusses the framework's features, advantages of using schemas in Spark SQL, and practical coding sessions. Additionally, the document addresses caching data and different data formats for reading and writing within Spark, emphasizing efficiency and productivity for developers and data scientists.
More Related Content
What's hot (20)
Watching Pigs Fly with the Netflix Hadoop Toolkit (Hadoop Summit 2013)
Watching Pigs Fly with the Netflix Hadoop Toolkit (Hadoop Summit 2013)Jeff Magnusson The document discusses Netflix's Hadoop toolkit presented at the Hadoop Summit 2013, focusing on making data accessible and processable using various tools like Franklin, Sting, and Lipstick. It highlights the importance of data visualization and user empowerment in creating effective recommendations based on user preferences and content similarity. Additionally, it invites feedback and contributions to their open-source projects and provides links to related resources.
An excursion into Graph Analytics with Apache Spark GraphX
An excursion into Graph Analytics with Apache Spark GraphXKrishna Sankar The document discusses the use of Apache Spark's GraphX for graph processing, outlining its API landscape, algorithms, and real-world applications. It includes a tutorial agenda covering topics such as graph structures, community analysis, and case studies like the AlphaGo tweet analysis. Additionally, it highlights various graph processing frameworks and partitioning strategies to efficiently manage large datasets.
Solr for Data Science
Solr for Data ScienceGrant Ingersoll The document discusses Solr, an open-source search platform with over 8 million downloads, highlighting its key features such as full-text search, data analysis tools, and massive scalability. It emphasizes the importance of quickly identifying significant data for exploration and includes information on integrations with tools like Logstash and Kibana. Lucidworks, the company behind Solr, is noted for its substantial contributions to Solr's development, providing various resources and integration options for data science applications.
Pandas/Data Analysis at Baypiggies
Pandas/Data Analysis at BaypiggiesAndy Hayden The document discusses lessons learned in the performance and design of the Python library Pandas, highlighting its benefits for data analysis and the importance of optimizing user experience. Key topics include user productivity, workflow improvements, API design considerations, and performance tracking using tools like vbench. The authors emphasize the need for robust documentation and careful optimization to ensure a seamless and efficient user experience.
Scalable Monitoring Using Apache Spark and Friends with Utkarsh Bhatnagar
Scalable Monitoring Using Apache Spark and Friends with Utkarsh BhatnagarDatabricks The document discusses the challenges and requirements for scalable monitoring using Apache Spark and Grafana, highlighting the need for handling millions of unique metrics and billions of data points daily. It details specific requirements for various teams, addressing the architectural considerations for scaling infrastructure and supporting real-time monitoring. Additionally, it mentions past achievements, the capability to process large volumes of metrics and logs, and invites for collaboration and hiring.
Jake Mannix, Lead Data Engineer, Lucidworks at MLconf SEA - 5/20/16
Jake Mannix, Lead Data Engineer, Lucidworks at MLconf SEA - 5/20/16MLconf The document outlines a practical data science workbench using Spark and Solr, focusing on data engineering and machine learning applications. It discusses key features and workflow examples, including data exploration, classification, and building recommender systems. The integration of Spark with Solr enhances capabilities for processing large datasets efficiently while providing powerful analytical tools.
sparklyr - Jeff Allen
sparklyr - Jeff AllenSri Ambati The document discusses sparklyr, an open-source R package that facilitates the integration of the R programming language with Apache Spark for big data analysis. It highlights the features of sparklyr, including full dplyr support and integration with RStudio, as well as its capabilities in managing bigger-than-memory data. Additionally, it compares sparklyr with the existing sparkr and introduces the rsparkling extension for H2O integration.
Taking Jupyter Notebooks and Apache Spark to the Next Level PixieDust with Da...
Taking Jupyter Notebooks and Apache Spark to the Next Level PixieDust with Da...Databricks The document discusses the integration of developers into data science processes using tools like Jupyter Notebooks and Pixiedust for building data-driven applications, notably for performing real-time sentiment analysis with Apache Spark. It highlights the collaboration between software developers and data scientists to address data challenges, emphasizing that future solutions depend on both roles working together. Key features of Pixiedust, including visualizations and user-friendly interfaces for non-technical users, are presented as essential elements for democratizing data analysis.
Hadoop for Data Science
Hadoop for Data ScienceDonald Miner The document discusses the application of Hadoop in data science, highlighting its distributed file system (HDFS) and MapReduce framework for efficient data analysis. It features examples and cool aspects of Hadoop including linear scalability, schema on read, and its capability to handle unstructured data. Various data science tasks such as data exploration, classification, natural language processing, and recommender systems are covered, along with the importance of using appropriate tools for different phases of model training workflows.
Deduplication and Author-Disambiguation of Streaming Records via Supervised M...
Deduplication and Author-Disambiguation of Streaming Records via Supervised M...Spark Summit The document discusses advancements in deduplication and author disambiguation of scholarly records using supervised models and content encoders within the Elsevier platform, emphasizing the importance of efficient data processing techniques. It outlines the challenges and solutions for managing peer-reviewed publications in Scopus, focusing on machine learning and cloud scalability for real-time document analysis. The implementation of algorithms like word2vec for document encoding aims to improve recommendation systems and personalized user experiences across various academic entities.
From R Script to Production Using rsparkling with Navdeep Gill
From R Script to Production Using rsparkling with Navdeep GillDatabricks The document presents an overview of H2O.ai, a company specializing in open-source software for machine learning and data science, including its key components such as H2O Sparkling Water, Sparklyr, and Rsparkling. It explains the integration of H2O within the Spark ecosystem, demonstrating how users can leverage advanced machine learning algorithms in their Spark workflows using these packages. The document also includes an example use case for R and Spark integration, providing a practical demonstration of the technologies discussed.
Fulfilling Apache Arrow's Promises: Pandas on JVM memory without a copy
Fulfilling Apache Arrow's Promises: Pandas on JVM memory without a copyUwe Korn This document discusses how Apache Arrow enables sharing data between Python and Java without copying. It summarizes Arrow's capabilities for efficient in-memory columnar data and its ability to exchange data between different programming languages. The document then outlines how Arrow, through its Java and Python libraries, allows querying data in Java from Python without copying, by passing memory addresses between the two environments. This enables faster data science workflows that involve both Python and Java/Scala.
Indexing big data in the cloud
Indexing big data in the cloudOpenSource Connections The document discusses strategies for indexing big data in the cloud, highlighting the importance of agility, minimalism, and effective use of tools like Solr. It outlines a real project approach, emphasizing the need for quick prototypes and thorough testing while managing large datasets. Key technical details on server setup and scaling methods are also included, along with the potential alternatives for managing cloud resources.
The Little Warehouse That Couldn't Or: How We Learned to Stop Worrying and Mo...
The Little Warehouse That Couldn't Or: How We Learned to Stop Worrying and Mo...Spark Summit The document discusses the challenges faced by Shopify in using its existing data warehouse and ETL processes due to increasing data volume and complexity. It describes Shopify's attempts to use Pig and Luigi as well as Platfora to address these issues, but notes they did not meet Shopify's needs. Shopify then moved to using Spark due to its fast performance, nice development model using Python, and ability to better handle their data and query complexity. The summary provides an overview of why Shopify changed its data warehousing approach and the key technology it adopted.
Large Scale Image Forensics using Tika and Tensorflow [ICMR MFSec 2017]
Large Scale Image Forensics using Tika and Tensorflow [ICMR MFSec 2017]Thamme Gowda The document presents a methodology for large-scale image forensics using deep learning, focused on monitoring illegal weapons sales through the DARPA Memex program. It describes the integration of TensorFlow with Apache Tika for analyzing 1.4 million weapon images using the Inception-v3 model, optimized for efficient performance. The evaluation indicates successful image recognition, aiding the classification of weapons and enhancing data retrieval capabilities.
Rental Cars and Industrialized Learning to Rank with Sean Downes
Rental Cars and Industrialized Learning to Rank with Sean DownesDatabricks The document discusses the challenges and principles of industrializing data science workflows in cloud infrastructure, highlighting the need for clarity around data ownership and organization. It presents idiosyncratic operating principles that emphasize the distinct roles of engineers, product managers, and data scientists. The presentation also stresses the importance of fostering a data-centric culture and closing the loop on data utilization.
Congressional PageRank: Graph Analytics of US Congress With Neo4j
Congressional PageRank: Graph Analytics of US Congress With Neo4jWilliam Lyon The document discusses modeling the US Congress as a graph database using Neo4j and analyzing relationships between legislators to identify influential members. It describes loading congressional data into Neo4j, querying relationships between legislators and states they represent. Methods for identifying influential legislators include degree centrality, betweenness centrality, and PageRank computed on a bill co-sponsorship graph using both Neo4j and Apache Spark with GraphX.
Extracting Insights from Data at Twitter
Extracting Insights from Data at TwitterPrasad Wagle Prasad Wagle's talk discussed how Twitter extracts insights from its large volumes of data. Twitter collects hundreds of millions of tweets and interactions per day from over 300 million monthly active users, creating big data challenges around velocity, volume, and variety. Twitter stores this data in hundreds of petabytes across large Hadoop clusters and processes it using batch tools like Hadoop and Spark as well as real-time tools like Heron. Insights are generated through basic analytics like user counts, A/B testing of new features, and custom data science work including machine learning models for recommendations, content filtering, and ad targeting. Systems, programming, and statistical skills are needed to effectively extract value from Twitter's big data.
PyCon.DE / PyData Karlsruhe keynote: "Looking backward, looking forward"
PyCon.DE / PyData Karlsruhe keynote: "Looking backward, looking forward"Wes McKinney This document discusses the history and development of Python data analysis tools, including pandas. It covers Wes McKinney's work on pandas from 2008 to the present, including the motivations for making data analysis easier and more productive. It also summarizes the development of related projects like Apache Arrow for standardizing columnar data representations to improve code reuse across languages.
Converging Big Data and Application Infrastructure by Steven Poutsy
Converging Big Data and Application Infrastructure by Steven PoutsyBig Data Spain The document introduces the evolution of application infrastructure, highlighting advancements such as distributed software, easier programming languages, and improved statistical libraries. It emphasizes the integration of big data, APIs, and machine learning with web applications using tools like Docker, Kubernetes, and OpenShift. The document provides resources for setting up OpenShift and related tools for analysis and encourages contributions through pull requests.
Viewers also liked (20)
Spark meetup v2.0.5
Spark meetup v2.0.5Yan Zhou The document discusses Huawei Technologies' implementation of Spark SQL on HBase, highlighting its motivations, performance capabilities, and future roadmap. It details the unique data challenges in telecommunications and how the technology can handle complex query types efficiently. Additionally, the paper reviews performance metrics, optimizations, and contributions to the Spark ecosystem from Huawei's big data teams.
DataEngConf SF16 - Spark SQL Workshop
DataEngConf SF16 - Spark SQL WorkshopHakka Labs The document outlines an agenda for an Apache Spark workshop, including topics such as Spark SQL, RDD operations, and user-defined functions (UDFs). It discusses the framework's features, advantages of using schemas in Spark SQL, and practical coding sessions. Additionally, the document addresses caching data and different data formats for reading and writing within Spark, emphasizing efficiency and productivity for developers and data scientists.
Spark Sql for Training
Spark Sql for TrainingBryan Yang The document covers an introduction to Spark SQL, including the use of SQLContext and HiveContext for data analysis. It provides details on creating DataFrames from RDDs and JSON, various DataFrame operations like select, filter, and group by, as well as saving data. Additionally, it discusses user-defined functions and data types supported in Spark SQL.
SparkSQL et Cassandra - Tool In Action Devoxx 2015
SparkSQL et Cassandra - Tool In Action Devoxx 2015Alexander DEJANOVSKI This document discusses the integration and use of Spark SQL with Cassandra for data analysis, highlighting key concepts like Resilient Distributed Datasets (RDDs) and various stages of data processing. It includes practical code snippets for SQL queries, data transformations in Scala and Python, and the use of DataStax connectors. The overall goal is to analyze and manage conference talk data from Devoxx using efficient techniques that leverage both Spark and Cassandra capabilities.
The SparkSQL things you maybe confuse
The SparkSQL things you maybe confusevito jeng SparkSQL, SchemaRDD, DataFrame, and Dataset are Apache Spark APIs for structured data processing. SparkSQL is a high-level module introduced in Spark 1.0. SchemaRDD was introduced in Spark 1.0 from the Shark project and was later renamed to DataFrame in Spark 1.3. Dataset, introduced experimentally in Spark 1.6, allows SparkSQL optimizations while working with RDDs. DataFrame and Dataset were unified under a single API in Spark 2.0.
Getting started with SparkSQL - Desert Code Camp 2016
Getting started with SparkSQL - Desert Code Camp 2016clairvoyantllc The document discusses Spark SQL, an Apache Spark module for structured data processing. It provides an agenda that covers Spark concepts, Spark SQL, the Catalyst optimizer, Project Tungsten, and a demo. Spark SQL allows users to perform SQL queries and use the DataFrame and Dataset APIs to interact with structured data in a Spark cluster.
Galvanise NYC - Scaling R with Hadoop & Spark. V1.0
Galvanise NYC - Scaling R with Hadoop & Spark. V1.0vithakur The document discusses scaling R using Hadoop and Spark. It provides an overview of IBM's approach to big data, which leverages open source technologies like Hadoop, Spark, and R. It then summarizes IBM's investments in Spark and the Open Data Platform initiative. The rest of the document focuses on describing Big R, IBM's tool for scaling R to big data using Hadoop. Big R allows users to run R scripts on large datasets in Hadoop and provides functions for machine learning algorithms and accessing Hadoop data from within R.
HBaseConEast2016: HBase and Spark, State of the Art
HBaseConEast2016: HBase and Spark, State of the ArtMichael Stack The document discusses the integration of HBase with Spark, highlighting improvements in data processing techniques and the transition from MapReduce to Spark Streaming. It outlines various Spark operations, including bulk loading and data retrieval from HBase, while emphasizing performance enhancements like data locality and partition pruning. Additionally, it touches on the development of DataFrame support, facilitating easier interactions between Spark and HBase.
The DAP - Where YARN, HBase, Kafka and Spark go to Production
The DAP - Where YARN, HBase, Kafka and Spark go to ProductionDataWorks Summit/Hadoop Summit The document summarizes the Cask Data Application Platform (CDAP), which provides an integrated framework for building and running data applications on Hadoop and Spark. It consolidates the big data application lifecycle by providing dataset abstractions, self-service data, metrics and log collection, lineage, audit, and access control. CDAP has an application container architecture with reusable programming abstractions and global user and machine metadata. It aims to simplify deploying and operating big data applications in enterprises by integrating technologies like YARN, HBase, Kafka and Spark.
The Nitty Gritty of Advanced Analytics Using Apache Spark in Python
The Nitty Gritty of Advanced Analytics Using Apache Spark in PythonMiklos Christine The document outlines the evolution and features of Apache Spark, highlighting its capabilities in advanced analytics and machine learning. It details Spark's architecture, usage statistics, best practices for ETL processes, and challenges like the small files problem. Additionally, it emphasizes the importance of using built-in functions and pipelines in Spark ML for efficient model building and data processing.
SparkR - Play Spark Using R (20160909 HadoopCon)
SparkR - Play Spark Using R (20160909 HadoopCon)wqchen The document provides a comprehensive overview of 'sparkr', an R interface for Apache Spark, highlighting its origins, features, and usage. It covers key concepts such as data frames, RDDs, machine learning libraries, and practical applications for data analysis and visualization using R. Additionally, it discusses the future development plans for sparkr and includes various references and resources for further learning.
Aioug vizag oracle12c_new_features
Aioug vizag oracle12c_new_featuresAiougVizagChapter The document summarizes new features in Oracle Database 12c from Oracle 11g that would help a DBA currently using 11g. It lists and briefly describes features such as the READ privilege, temporary undo, online data file move, DDL logging, and many others. The objectives are to make the DBA aware of useful 12c features when working with a 12c database and to discuss each feature at a high level within 90 seconds.
COUG_AAbate_Oracle_Database_12c_New_Features
COUG_AAbate_Oracle_Database_12c_New_FeaturesAlfredo Abate Oracle Database 12c includes over 500 new features. Some key new features include:
- Oracle Database 12c Express (EM Express) which replaces Database Control and has less features than Database Control but does not require Java or an app server.
- New online capabilities like online DDL operations with no DDL locking, online move of partitions with no impact to queries, and online statistics gathering for bulk loads.
- Adaptive SQL Plan Management which allows the optimizer to select a more optimal plan at execution time based on current statistics.
- Multitenant architecture which allows consolidation of multiple databases into one container database with pluggable databases.
Oracle12 - The Top12 Features by NAYA Technologies
Oracle12 - The Top12 Features by NAYA TechnologiesNAYATech The document discusses the top 12 new features of Oracle 12c, as presented by David Yahalom of NAYA Technologies. It covers improved column defaults, increased size limits, improved top-N queries, temporary UNDO, new partitioning features, transaction guard, adaptive execution plans, enhanced statistics, data optimization and information lifecycle management (ILM), row pattern matching, and a 50% discount code for a Oracle performance tuning seminar offered by NAYA Technologies.
Petabyte Scale Anomaly Detection Using R & Spark by Sridhar Alla and Kiran Mu...
Petabyte Scale Anomaly Detection Using R & Spark by Sridhar Alla and Kiran Mu...Spark Summit This document discusses using Spark and R for petabyte-scale data science at Comcast. It describes Comcast's top data initiatives, how Spark and SparkR enable machine learning algorithms, and an example of using a Hidden Markov Model with SparkR to analyze streaming data and detect hidden states. Performance testing showed the model processing 1.7 billion observations per day in 30 minutes with 92% accuracy.
Build a Time Series Application with Apache Spark and Apache HBase
Build a Time Series Application with Apache Spark and Apache HBaseCarol McDonald This document discusses using Apache Spark and Apache HBase to build a time series application. It provides an overview of time series data and requirements for ingesting, storing, and analyzing high volumes of time series data. The document then describes using Spark Streaming to process real-time data streams from sensors and storing the data in HBase. It outlines the steps in the lab exercise, which involves reading sensor data from files, converting it to objects, creating a Spark Streaming DStream, processing the DStream, and saving the data to HBase.
Introduce to Spark sql 1.3.0
Introduce to Spark sql 1.3.0 Bryan Yang This document introduces Spark SQL 1.3.0 and how to optimize efficiency. It discusses the main objects like SQL Context and how to create DataFrames from RDDs, JSON, and perform operations like select, filter, groupBy, join, and save data. It shows how to register DataFrames as tables and write SQL queries. DataFrames also support RDD actions and transformations. The document provides references for learning more about DataFrames and their development direction.
Spark etl
Spark etlImran Rashid The document provides tips for writing effective ETL pipelines using Apache Spark, emphasizing the importance of modularity, error handling, and understanding performance. Key concepts such as RDDs, error catching with accumulators, and best practices for data partitioning and transformation are discussed. Additionally, the document highlights the necessity of minimizing data volume and monitoring stage boundaries for optimizing Spark applications.
Apache HBase Internals you hoped you Never Needed to Understand
Apache HBase Internals you hoped you Never Needed to UnderstandJosh Elser The document outlines the internals of Apache HBase, detailing its architecture, region management, and data handling processes. It further discusses the role of Apache ZooKeeper in coordination and resiliency within distributed systems, and highlights features such as writes, compactions, and disaster recovery mechanisms. Additionally, it addresses security aspects, including Kerberos integration for authentication in HBase environments.
SPARQL and Linked Data Benchmarking
SPARQL and Linked Data BenchmarkingKristian Alexander The document discusses various benchmarks that are commonly used to evaluate Semantic Web repositories and their performance handling large amounts of RDF data. Some of the major benchmarks mentioned include the Lehigh University Benchmark (LUBM), Berlin SPARQL Benchmark (BSBM), SP2Bench, Social Network Intelligence Benchmark (SIB), and DBPedia SPARQL Benchmark. The document also provides an overview of different benchmark components and links to resources with performance results from various RDF stores and systems.
Ad
Similar to Data Science at Scale: Using Apache Spark for Data Science at Bitly (20)
Data Science at Scale by Sarah Guido
Data Science at Scale by Sarah GuidoSpark Summit This document summarizes Sarah Guido's talk on using Apache Spark for data science at Bitly. She discusses how Bitly uses Spark to extract, explore, and model subsets of their data including decoding Bitly links, performing topic modeling using LDA, and trend detection. While Spark provides performance benefits over MapReduce for these tasks, she notes issues with Hadoop servers, JVM, and lack of documentation that must be addressed for full production usage at Bitly.
Apache Spark in Industry
Apache Spark in IndustryDorian Beganovic 1. Apache Spark is an open source cluster computing framework for large-scale data processing. It is compatible with Hadoop and provides APIs for SQL, streaming, machine learning, and graph processing.
2. Over 3000 companies use Spark, including Microsoft, Uber, Pinterest, and Amazon. It can run on standalone clusters, EC2, YARN, and Mesos.
3. Spark SQL, Streaming, and MLlib allow for SQL queries, streaming analytics, and machine learning at scale using Spark's APIs which are inspired by Python/R data frames and scikit-learn.
Deep Learning on Apache® Spark™ : Workflows and Best Practices
Deep Learning on Apache® Spark™ : Workflows and Best PracticesJen Aman The document presents an overview of deep learning workflows and best practices using Apache Spark, highlighting its integration with various deep learning frameworks such as TensorFlow and Caffe. It discusses the architecture of Databricks, which simplifies data processing and machine learning for users, as well as monitoring techniques and solutions for managing distributed deep learning tasks. The presentation emphasizes the importance of experimentation and provides resources for further learning.
Deep Learning on Apache® Spark™: Workflows and Best Practices
Deep Learning on Apache® Spark™: Workflows and Best PracticesDatabricks The document discusses deep learning integration with Apache Spark, emphasizing workflows and best practices. It highlights the various deep learning frameworks supported by Spark, including TensorFlow and Keras, and provides insights into data pipeline patterns, monitoring, and the use of GPUs. The talk aims to guide users in optimizing their deep learning applications within the Spark ecosystem.
Deep Learning on Apache® Spark™: Workflows and Best Practices
Deep Learning on Apache® Spark™: Workflows and Best PracticesJen Aman This document summarizes a presentation about deep learning workflows and best practices on Apache Spark. It discusses how deep learning fits within broader data pipelines for tasks like training and transformation. It also outlines recurring patterns for integrating Spark and deep learning frameworks, including using Spark for data parallelism and embedding deep learning transforms. The presentation provides tips for developers on topics like using GPUs with PySpark and monitoring deep learning jobs. It concludes by discussing challenges in the areas of distributed deep learning and Spark integration.
Spark SQL
Spark SQLCaserta The document provides an agenda and overview for a Big Data Warehousing meetup hosted by Caserta Concepts. The meetup agenda includes an introduction to SparkSQL with a deep dive on SparkSQL and a demo. Elliott Cordo from Caserta Concepts will provide an introduction and overview of Spark as well as a demo of SparkSQL. The meetup aims to share stories in the rapidly changing big data landscape and provide networking opportunities for data professionals.
Apache Spark Fundamentals
Apache Spark FundamentalsZahra Eskandari This document provides an overview of the Apache Spark framework. It covers Spark fundamentals including the Spark execution model using Resilient Distributed Datasets (RDDs), basic Spark programming, and common Spark libraries and use cases. Key topics include how Spark improves on MapReduce by operating in-memory and supporting general graphs through its directed acyclic graph execution model. The document also reviews Spark installation and provides examples of basic Spark programs in Scala.
Big Data Processing with Apache Spark 2014
Big Data Processing with Apache Spark 2014mahchiev This document provides an overview of Apache Spark, a framework for large-scale data processing. It discusses what big data is, the history and advantages of Spark, and Spark's execution model. Key concepts explained include Resilient Distributed Datasets (RDDs), transformations, actions, and MapReduce algorithms like word count. Examples are provided to illustrate Spark's use of RDDs and how it can improve on Hadoop MapReduce.
Big_data_analytics_NoSql_Module-4_Session
Big_data_analytics_NoSql_Module-4_SessionRUHULAMINHAZARIKA - Apache Spark is an open-source cluster computing framework that is faster than Hadoop for batch processing and also supports real-time stream processing.
- Spark was created to be faster than Hadoop for interactive queries and iterative algorithms by keeping data in-memory when possible.
- Spark consists of Spark Core for the basic RDD API and also includes modules for SQL, streaming, machine learning, and graph processing. It can run on several cluster managers including YARN and Mesos.
Architecting Your First Big Data Implementation
Architecting Your First Big Data ImplementationAdaryl "Bob" Wakefield, MBA This document provides an overview of architecting a first big data implementation. It defines key concepts like Hadoop, NoSQL databases, and real-time processing. It recommends asking questions about data, technology stack, and skills before starting a project. Distributed file systems, batch tools, and streaming systems like Kafka are important technologies for big data architectures. The document emphasizes moving from batch to real-time processing as a major opportunity.
From a student to an apache committer practice of apache io tdb
From a student to an apache committer practice of apache io tdbjixuan1989 The document describes the experiences of Xiangdong Huang, a PhD student from Tsinghua University, in developing Apache IoTDB, an open-source time series database. It outlines his journey from early coding experiences to significant involvement in an open-source community, highlighting challenges and learning opportunities in software development. The document emphasizes teamwork, skills development, and the goals of the Apache IoTDB project, which aims to provide an efficient database for managing time series data in industrial applications.
Apache Spark - Lightning Fast Cluster Computing - Hyderabad Scalability Meetup
Apache Spark - Lightning Fast Cluster Computing - Hyderabad Scalability MeetupHyderabad Scalability Meetup This document provides an introduction and overview of Apache Spark, a lightning-fast cluster computing framework. It discusses Spark's ecosystem, how it differs from Hadoop MapReduce, where it shines well, how easy it is to install and start learning, includes some small code demos, and provides additional resources for information. The presentation introduces Spark and its core concepts, compares it to Hadoop MapReduce in areas like speed, usability, tools, and deployment, demonstrates how to use Spark SQL with an example, and shows a visualization demo. It aims to provide attendees with a high-level understanding of Spark without being a training class or workshop.
Dec6 meetup spark presentation
Dec6 meetup spark presentationRamesh Mudunuri This document provides an introduction and overview of Apache Spark, a lightning-fast cluster computing framework. It discusses Spark's ecosystem, how it differs from Hadoop MapReduce, where it shines well, how easy it is to install and start learning, includes some small code demos, and provides additional resources for information. The presentation introduces Spark and its core concepts, compares it to Hadoop MapReduce in areas like speed, usability, tools, and deployment, demonstrates how to use Spark SQL with an example, and shows a visualization demo. It aims to provide attendees with a high-level understanding of Spark without being a training class or workshop.
Big Data Introduction - Solix empower
Big Data Introduction - Solix empowerDurga Gadiraju This document provides an overview of big data and the Spark framework. It discusses the big data ecosystem, including file systems, data ingestion tools, batch and real-time data processing frameworks, visualization tools, and support technologies. It outlines common big data job roles and their associated skills. The document then focuses on Spark, describing its core functionality, modules like DataFrames and MLlib, and execution modes. It provides guidance on learning Spark, emphasizing programming skills and Spark APIs. A demo of Spark fundamentals on a big data lab is also proposed.
Scala and Spark are Ideal for Big Data - Data Science Pop-up Seattle
Scala and Spark are Ideal for Big Data - Data Science Pop-up SeattleDomino Data Lab The document discusses the advantages of using Scala and Spark for big data applications, highlighting Scala's strong typing and concise syntax, as well as its compatibility with functional programming. It emphasizes Spark's performance benefits, such as real-time processing, fast in-memory data access, and its scalability for large datasets. The document also includes resources for learning Scala and Spark through meetups and training programs in Seattle.
Hadoop Ecosystem
Hadoop EcosystemLior Sidi The document provides an overview of the Hadoop ecosystem, detailing its components such as HDFS for storage, YARN for resource management, and MapReduce for processing large datasets. It discusses the evolution of Hadoop, the challenges it addresses, and includes various data integration tools like Sqoop and Flume, as well as streaming options like Kafka. Additionally, it covers the role of Apache projects such as Hive and Spark in data analysis and visualization within the Hadoop framework.
Hadoop world overview trends and topics
Hadoop world overview trends and topicsValentin Kropov The document outlines a conference agenda focusing on advancements in big data technologies such as Hadoop map-reduce and Apache Spark, highlighting Spark's efficiency and ease of use. It discusses Cloudera's Kudu offering for low-latency data access and the development of popular reference architectures involving technologies like YARN, Kafka, and Cassandra. Additionally, it shares case studies on Netflix's big data platform and NASA's Orion mission, emphasizing data storage and analytics capabilities.
Spark Uber Development Kit
Spark Uber Development KitDataWorks Summit/Hadoop Summit The document discusses tools and techniques used by Uber's Hadoop team to make their Spark and Hadoop platforms more user-friendly and efficient. It introduces tools like SCBuilder to simplify Spark context creation, Kafka dispersal to distribute RDD results, and SparkPlug to provide templates for common jobs. It also describes a distributed log debugger called SparkChamber to help debug Spark jobs and techniques like building a spatial index to optimize geo-spatial joins. The goal is to abstract out infrastructure complexities and enforce best practices to make the platforms more self-service for users.
Machine Learning at Scale
Machine Learning at ScaleMadhukara Phatak This document summarizes Madhukara Phatak's journey working with machine learning and big data technologies. It describes his work with Hadoop, Mahout, JavaScript, Scala, Spark, MLLib and building a rumor engine application. Some key points include:
- He encountered challenges with Mahout's performance and complexity which led him to explore Spark and build his own libraries.
- Spark provided better support for iterative algorithms, caching, and was more productive for machine learning compared to MapReduce.
- His rumor engine application classified blog posts using Naive Bayes with MLLib and had high performance on Spark.
- Finding skilled data scientists remains a challenge due to the unique combination of skills required.
Intro to Apache Spark by CTO of Twingo
Intro to Apache Spark by CTO of TwingoMapR Technologies This document provides an overview of Apache Spark, highlighting its features as a fast and general data processing engine that outperforms Hadoop through in-memory data storage and a unified platform for various analytics. It covers Spark's ease of development in languages such as Java, Scala, and Python, as well as its applications in machine learning and streaming. The document also mentions real-world use cases and resources for leveraging Spark in big data projects.
Apache Spark - Lightning Fast Cluster Computing - Hyderabad Scalability Meetup
Apache Spark - Lightning Fast Cluster Computing - Hyderabad Scalability MeetupHyderabad Scalability Meetup
Ad
More from Sarah Guido (7)
Data Science Retrospective
Data Science RetrospectiveSarah Guido The document summarizes how data science has changed over the past 6 years based on the presenter's experience. Some key points that have remained the same are the continued dominance of Python and messy real-world data. Things that have changed include increased specialization within roles, more remote opportunities, improved tools and infrastructure for working with big data, and greater availability of learning resources. While the hype around data science fluctuates, the core skills remain similar with an emphasis on programming, statistics, communication and independence.
The Wild West of Data Wrangling (PyTN)
The Wild West of Data Wrangling (PyTN)Sarah Guido This talk discusses techniques for dealing with messy and imperfect data. Three examples are provided: (1) commercial real estate data with validity issues that were addressed through consensus, imputation, outlier removal and OCR; (2) click data with cookie issues that required transformations like sampling and clustering after preprocessing; (3) digital media data with access issues that necessitated working with insufficient data using decomposition and seeking more data. The overall message is to focus on the problem, identify data limitations, and apply creative solutions like reworking the data.
The Wild West of Data Wrangling
The Wild West of Data WranglingSarah Guido This document summarizes a talk by Sarah Guido at PyCon 2017, focusing on challenges in data wrangling and various examples from her work as a senior data scientist at Mashable. It discusses predicting building sales using logistic regression and addressing class imbalance, clustering user interactions with k-means, and understanding audience composition despite data limitations. The talk emphasizes the importance of problem identification, data quality, and creative solutions in data science.
The Importance of Community
The Importance of CommunitySarah Guido The document discusses the importance of getting involved in the Python community. It describes the speaker's journey learning Python and data science through attending conferences and networking. The speaker recommends strategies for avoiding burnout like collaborating, asking for help, and taking breaks. Getting involved provides opportunities to learn new skills, meet people, connect with potential employers and employees, gain help with projects, and boost confidence. Ways to participate include attending local meetups, conferences, contributing to open source, co-presenting, and occasionally declining opportunities to avoid burnout. Resources are provided for finding existing meetup groups and tips for starting a new one.
Network theory - PyCon 2015
Network theory - PyCon 2015Sarah Guido The document presents a network analysis of Twitter using the NetworkX library, detailing the concepts of network theory and graph representation. It covers key topics such as nodes, edges, centrality measures (degree, betweenness, and closeness), and different types of network models including random, small world, and scale-free networks. The speakers, Sarah Guido and Celia La, provide their backgrounds as data scientists and software engineers, while also emphasizing practical lessons and hands-on approaches to exploring network data.
K-means Clustering with Scikit-Learn
K-means Clustering with Scikit-LearnSarah Guido This document provides an overview of k-means clustering using the scikit-learn library, including its definition, advantages, and disadvantages. It outlines the iterative process of clustering, selecting parameters, and evaluating results using silhouette scores. Additionally, it offers tips for effective data exploration and resources for further learning.
A Beginner's Guide to Machine Learning with Scikit-Learn
A Beginner's Guide to Machine Learning with Scikit-LearnSarah Guido The document is a beginner's guide to machine learning using scikit-learn, covering supervised and unsupervised learning, data preprocessing, and validation strategies. It provides an overview of important machine learning concepts, vocabulary, and practical applications in data analysis. Additionally, it offers resources for learning scikit-learn, including documentation and tutorials.
Recently uploaded (20)
You are not excused! How to avoid security blind spots on the way to production
You are not excused! How to avoid security blind spots on the way to productionMichele Leroux Bustamante We live in an ever evolving landscape for cyber threats creating security risk for your production systems. Mitigating these risks requires participation throughout all stages from development through production delivery - and by every role including architects, developers QA and DevOps engineers, product owners and leadership. No one is excused! This session will cover examples of common mistakes or missed opportunities that can lead to vulnerabilities in production - and ways to do better throughout the development lifecycle.
Lessons Learned from Developing Secure AI Workflows.pdf
Lessons Learned from Developing Secure AI Workflows.pdfPriyanka Aash Lessons Learned from Developing Secure AI Workflows
PyCon SG 25 - Firecracker Made Easy with Python.pdf
PyCon SG 25 - Firecracker Made Easy with Python.pdfMuhammad Yuga Nugraha Explore the ease of managing Firecracker microVM with the firecracker-python. In this session, I will introduce the basics of Firecracker microVM and demonstrate how this custom SDK facilitates microVM operations easily. We will delve into the design and development process behind the SDK, providing a behind-the-scenes look at its creation and features. While traditional Firecracker SDKs were primarily available in Go, this module brings a simplicity of Python to the table.
The Future of Product Management in AI ERA.pdf
The Future of Product Management in AI ERA.pdfAlyona Owens Hi, I’m Aly Owens, I have a special pleasure to stand here as over a decade ago I graduated from CityU as an international student with an MBA program. I enjoyed the diversity of the school, ability to work and study, the network that came with being here, and of course the price tag for students here has always been more affordable than most around.
Since then I have worked for major corporations like T-Mobile and Microsoft and many more, and I have founded a startup. I've also been teaching product management to ensure my students save time and money to get to the same level as me faster avoiding popular mistakes. Today as I’ve transitioned to teaching and focusing on the startup, I hear everybody being concerned about Ai stealing their jobs… We’ll talk about it shortly.
But before that, I want to take you back to 1997. One of my favorite movies is “Fifth Element”. It wowed me with futuristic predictions when I was a kid and I’m impressed by the number of these predictions that have already come true. Self-driving cars, video calls and smart TV, personalized ads and identity scanning. Sci-fi movies and books gave us many ideas and some are being implemented as we speak. But we often get ahead of ourselves:
Flying cars,Colonized planets, Human-like AI: not yet, Time travel, Mind-machine neural interfaces for everyone: Only in experimental stages (e.g. Neuralink).
Cyberpunk dystopias: Some vibes (neon signs + inequality + surveillance), but not total dystopia (thankfully).
On the bright side, we predict that the working hours should drop as Ai becomes our helper and there shouldn’t be a need to work 8 hours/day. Nobody knows for sure but we can require that from legislation. Instead of waiting to see what the government and billionaires come up with, I say we should design our own future.
So, we as humans, when we don’t know something - fear takes over. The same thing happened during the industrial revolution. In the Industrial Era, machines didn’t steal jobs—they transformed them but people were scared about their jobs. The AI era is making similar changes except it feels like robots will take the center stage instead of a human. First off, even when it comes to the hottest space in the military - drones, Ai does a fraction of work. AI algorithms enable real-time decision-making, obstacle avoidance, and mission optimization making drones far more autonomous and capable than traditional remote-controlled aircraft. Key technologies include computer vision for object detection, GPS-enhanced navigation, and neural networks for learning and adaptation. But guess what? There are only 2 companies right now that utilize Ai in drones to make autonomous decisions - Skydio and DJI.
Salesforce Summer '25 Release Frenchgathering.pptx.pdf
Salesforce Summer '25 Release Frenchgathering.pptx.pdfyosra Saidani Salesforce Summer '25 Release Frenchgathering.pptx.pdf
Coordinated Disclosure for ML - What's Different and What's the Same.pdf
Coordinated Disclosure for ML - What's Different and What's the Same.pdfPriyanka Aash Coordinated Disclosure for ML - What's Different and What's the Same
OpenACC and Open Hackathons Monthly Highlights June 2025
OpenACC and Open Hackathons Monthly Highlights June 2025OpenACC The OpenACC organization focuses on enhancing parallel computing skills and advancing interoperability in scientific applications through hackathons and training. The upcoming 2025 Open Accelerated Computing Summit (OACS) aims to explore the convergence of AI and HPC in scientific computing and foster knowledge sharing. This year's OACS welcomes talk submissions from a variety of topics, from Using Standard Language Parallelism to Computer Vision Applications. The document also highlights several open hackathons, a call to apply for NVIDIA Academic Grant Program and resources for optimizing scientific applications using OpenACC directives.
"How to survive Black Friday: preparing e-commerce for a peak season", Yurii ...
"How to survive Black Friday: preparing e-commerce for a peak season", Yurii ...Fwdays We will explore how e-commerce projects prepare for the busiest time of the year, which key aspects to focus on, and what to expect. We’ll share our experience in setting up auto-scaling, load balancing, and discuss the loads that Silpo handles, as well as the solutions that help us navigate this season without failures.
Curietech AI in action - Accelerate MuleSoft development
Curietech AI in action - Accelerate MuleSoft developmentshyamraj55 CurieTech AI in Action – Accelerate MuleSoft Development
Overview:
This presentation demonstrates how CurieTech AI’s purpose-built agents empower MuleSoft developers to create integration workflows faster, more accurately, and with less manual effort
linkedin.com
+12
curietech.ai
+12
meetups.mulesoft.com
+12
.
Key Highlights:
Dedicated AI agents for every stage: Coding, Testing (MUnit), Documentation, Code Review, and Migration
curietech.ai
+7
curietech.ai
+7
medium.com
+7
DataWeave automation: Generate mappings from tables or samples—95%+ complete within minutes
linkedin.com
+7
curietech.ai
+7
medium.com
+7
Integration flow generation: Auto-create Mule flows based on specifications—speeds up boilerplate development
curietech.ai
+1
medium.com
+1
Efficient code reviews: Gain intelligent feedback on flows, patterns, and error handling
youtube.com
+8
curietech.ai
+8
curietech.ai
+8
Test & documentation automation: Auto-generate MUnit test cases, sample data, and detailed docs from code
curietech.ai
+5
curietech.ai
+5
medium.com
+5
Why Now?
Achieve 10× productivity gains, slashing development time from hours to minutes
curietech.ai
+3
curietech.ai
+3
medium.com
+3
Maintain high accuracy with code quality matching or exceeding manual efforts
curietech.ai
+2
curietech.ai
+2
curietech.ai
+2
Ideal for developers, architects, and teams wanting to scale MuleSoft projects with AI efficiency
Conclusion:
CurieTech AI transforms MuleSoft development into an AI-accelerated workflow—letting you focus on innovation, not repetition.
EIS-Webinar-Engineering-Retail-Infrastructure-06-16-2025.pdf
EIS-Webinar-Engineering-Retail-Infrastructure-06-16-2025.pdfEarley Information Science As AI reshapes expectations in retail and B2B commerce, organizations are recognizing a critical reality: meaningful AI outcomes depend on well-structured, adaptable infrastructure. In this session, Seth Earley is joined by Phil Ryan - AI strategist, search technologist, and founder of Glass Leopard Technologies - for a candid conversation on what it truly means to engineer systems for scale, agility, and intelligence.
Phil draws on more than two decades of experience leading search and AI initiatives for enterprise organizations. Together, he and Seth explore the challenges businesses face when legacy architectures limit personalization, agility, and real-time decisioning - and what needs to change to support agentic technologies and next-best-action capabilities.
Key themes from the webinar include:
Composability as a prerequisite for AI - Why modular, loosely coupled systems are essential for adapting to rapid innovation and evolving business needs
Search and relevance as foundational to AI - How techniques honed-in enterprise search have laid the groundwork for more responsive and intelligent customer experiences
From MDM and CDP to agentic systems - How data platforms are evolving to support richer customer context and dynamic orchestration
Engineering for business alignment - Why successful AI programs require architectural decisions grounded in measurable outcomes
The conversation is practical and forward-looking, connecting deep technical understanding with real-world business needs. Whether you’re modernizing your commerce stack or exploring how AI can enhance product discovery, personalization, or customer journeys, this session provides a clear-eyed view of the capabilities, constraints, and priorities that matter most.
Enhance GitHub Copilot using MCP - Enterprise version.pdf
Enhance GitHub Copilot using MCP - Enterprise version.pdfNilesh Gule Slide deck related to the GitHub Copilot Bootcamp in Melbourne on 17 June 2025
From Manual to Auto Searching- FME in the Driver's Seat
From Manual to Auto Searching- FME in the Driver's SeatSafe Software Finding a specific car online can be a time-consuming task, especially when checking multiple dealer websites. A few years ago, I faced this exact problem while searching for a particular vehicle in New Zealand. The local classified platform, Trade Me (similar to eBay), wasn’t yielding any results, so I expanded my search to second-hand dealer sites—only to realise that periodically checking each one was going to be tedious. That’s when I noticed something interesting: many of these websites used the same platform to manage their inventories. Recognising this, I reverse-engineered the platform’s structure and built an FME workspace that automated the search process for me. By integrating API calls and setting up periodic checks, I received real-time email alerts when matching cars were listed. In this presentation, I’ll walk through how I used FME to save hours of manual searching by creating a custom car-finding automation system. While FME can’t buy a car for you—yet—it can certainly help you find the one you’re after!
Using the SQLExecutor for Data Quality Management: aka One man's love for the...
Using the SQLExecutor for Data Quality Management: aka One man's love for the...Safe Software The SQLExecutor is one of FME’s most powerful and flexible transformers. Pivvot maintains a robust internal metadata hierarchy used to support ingestion and curation of thousands of external data sources that must be managed for quality before entering our platform. By using the SQLExecutor, Pivvot can efficiently detect problems and perform analysis before data is extracted from our staging environment, removing the need for rollbacks or cycles waisted on a failed job. This presentation will walk through three distinct examples of how Pivvot uses the SQLExecutor to engage its metadata hierarchy and integrate with its Data Quality Management workflows efficiently and within the source postgres database. Spatial Validation –Validating spatial prerequisites before entering a production environment. Reference Data Validation - Dynamically validate domain-ed columns across any table and multiple columns per table. Practical De-duplication - Removing identical or near-identical well point locations from two distinct source datasets in the same table.
cnc-processing-centers-centateq-p-110-en.pdf
cnc-processing-centers-centateq-p-110-en.pdfAmirStern2 מרכז עיבודים תעשייתי בעל 3/4/5 צירים, עד 22 החלפות כלים עם כל אפשרויות העיבוד הדרושות. בעל שטח עבודה גדול ומחשב נוח וקל להפעלה בשפה העברית/רוסית/אנגלית/ספרדית/ערבית ועוד..
מסוגל לבצע פעולות עיבוד שונות המתאימות לענפים שונים: קידוח אנכי, אופקי, ניסור, וכרסום אנכי.
Tech-ASan: Two-stage check for Address Sanitizer - Yixuan Cao.pdf
Tech-ASan: Two-stage check for Address Sanitizer - Yixuan Cao.pdfcaoyixuan2019 A presentation at Internetware 2025.
9-1-1 Addressing: End-to-End Automation Using FME
9-1-1 Addressing: End-to-End Automation Using FMESafe Software This session will cover a common use case for local and state/provincial governments who create and/or maintain their 9-1-1 addressing data, particularly address points and road centerlines. In this session, you'll learn how FME has helped Shelby County 9-1-1 (TN) automate the 9-1-1 addressing process; including automatically assigning attributes from disparate sources, on-the-fly QAQC of said data, and reporting. The FME logic that this presentation will cover includes: Table joins using attributes and geometry, Looping in custom transformers, Working with lists and Change detection.
“MPU+: A Transformative Solution for Next-Gen AI at the Edge,” a Presentation...
“MPU+: A Transformative Solution for Next-Gen AI at the Edge,” a Presentation...Edge AI and Vision Alliance For the full video of this presentation, please visit: https://www.edge-ai-vision.com/2025/06/mpu-a-transformative-solution-for-next-gen-ai-at-the-edge-a-presentation-from-fotonation/
Petronel Bigioi, CEO of FotoNation, presents the “MPU+: A Transformative Solution for Next-Gen AI at the Edge” tutorial at the May 2025 Embedded Vision Summit.
In this talk, Bigioi introduces MPU+, a novel programmable, customizable low-power platform for real-time, localized intelligence at the edge. The platform includes an AI-augmented image signal processor that enables leading image and video quality.
In addition, it integrates ultra-low-power object and motion detection capabilities to enable always-on computer vision. A programmable neural processor provides flexibility to efficiently implement new neural networks. And additional specialized engines facilitate image stabilization and audio enhancements.
Raman Bhaumik - Passionate Tech Enthusiast
Raman Bhaumik - Passionate Tech EnthusiastRaman Bhaumik A Junior Software Developer with a flair for innovation, Raman Bhaumik excels in delivering scalable web solutions. With three years of experience and a solid foundation in Java, Python, JavaScript, and SQL, she has streamlined task tracking by 20% and improved application stability.
You are not excused! How to avoid security blind spots on the way to production
You are not excused! How to avoid security blind spots on the way to productionMichele Leroux Bustamante
“MPU+: A Transformative Solution for Next-Gen AI at the Edge,” a Presentation...
“MPU+: A Transformative Solution for Next-Gen AI at the Edge,” a Presentation...Edge AI and Vision Alliance
Data Science at Scale: Using Apache Spark for Data Science at Bitly
- 1. Data Science at Scale: Using Apache Spark for Data Science at Bitly Sarah Guido Data Day Seattle 2015
- 2. Overview • About me/Bitly • Spark overview • Using Spark for data science • When it works, it’s great! When it works…
- 3. About me • Data scientist at Bitly • NYC Python/PyGotham co-organizer • O’Reilly Media author • @sarah_guido
- 4. About this talk • This talk is: – Description of my workflow – Exploration of within-Spark tools • This talk is not: – In-depth exploration of algorithms – Building new tools on top of Spark – Any sort of ground truth for how you should be using Spark
- 5. A bit of background • Need for big data analysis tools • MapReduce for exploratory data analysis == • Iterate/prototype quickly • Overall goal: understand how people use not only our app, but the Internet!
- 6. Bitly data! • Legit big data • 1 hour of decodes is 10 GB • 1 day is 240 GB • 1 month is ~7 TB
- 7. What is Spark? • Large-scale distributed data processing tool • SQL and streaming tools • Faster than Hadoop • Python API
- 8. How does Spark work? • Partitions your data to operate over in parallel – A partition by default is 64 MB • Capability to add map/reduce features • Lazy – only operates when method is called – Ex. collect() or writing to a file
- 9. Why Spark? • Fast. Really fast. • SQL layer – kind of like Hive • Distributed scientific tools • Python! Sometimes. • Cutting edge technology
- 10. Setting up the workflow • Spark journey – Hadoop server: 1.2 – EMR: 1.3 – EMR: 1.4
- 11. How do I use it? • EMR! • spark-submit on the cluster • Can add script as a step to cluster launch
- 12. Creating a cluster • aws emr create-cluster • --bootstrap-action • --steps • --auto-terminate
- 13. Creating a cluster • LIVE!!
- 14. Let’s set the stage… • Understanding user behavior • How do I extract, explore, and model a subset of our data using Spark?
- 15. Data {"a": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_10_2) AppleWebKit/600.4.10 (KHTML, like Gecko) Version/8.0.4 Safari/600.4.10", "c": "US", "nk": 0, "tz": "America/Los_Angeles", "g": "1HfTjh8", "h": "1HfTjh7", "u": "http://www.nytimes.com/2015/03/22/opinion/sunday/why- health-care-tech-is-still-so-bad.html?smid=tw-share", "t": 1427288425, "cy": "Seattle"}
- 16. Data processing • Problem: I want to retrieve NYT decodes • Solution: well, there are two…
- 17. Data processing
- 18. Data processing
- 19. Data processing • SparkSQL: 8 minutes • Pure Spark: 4 minutes!!!
- 20. Data processing
- 21. Data processing • Yes, we’re going to do a live demo of this!
- 22. Exploratory data analysis • Problem: what’s going on with my decodes? • Solution: DataFrames! – Similar to Pandas: describe, drop, fill, aggregate functions – You can actually convert to a Pandas DataFrame!
- 23. Exploratory data analysis • Get a sense of what’s going on in the data • Look at distributions, frequencies • Mostly categorical data here
- 24. Exploratory data analysis • Yet another live demo
- 25. Topic modeling • Problem: we have so many links but no way to classify them into certain kinds of content • Solution: LDA (latent Dirichlet allocation) – Sort of – compare to other solutions
- 26. Topic modeling • Oh, the JVM… – LDA only in Scala • Scala jar file • Store script in S3
- 27. Topic modeling • LDA in Spark – Generative model – Several different methods – Term frequency vector as input • “Note: LDA is a new feature with some missing functionality...”
- 28. Topic modeling
- 29. Topic modeling • Term frequency vector TERM DOCUMENT python data hot dogs baseball zoo doc_1 1 3 0 0 0 doc_2 0 0 4 1 0 doc_3 4 0 0 0 5
- 30. Topic modeling
- 31. Topic modeling
- 32. Topic modeling • Why not?? – Means to an end – Current large scale scraping inability
- 33. Architecture • Right now: not in production – Buy-in • Streaming applications for parts of the app • Python or Scala? – Scala by force (LDA, GraphX)
- 34. Some issues • Hadoop servers • JVM • gzip • 1.4 • Resource allocation • Really only got it to this stage very recently
- 35. Where to go next? • Spark in production! • Use for various parts of our app • Use for R&D and prototyping purposes, with the potential to expand into the product
- 36. Current/future projects • Trend detection • Device prediction • User affinities – GraphX! • A/B testing
- 37. Resources • spark.apache.org - documentation • Databricks blog • Cloudera blog