Using Databricks as an Analysis Platform
3 likes1,012 views
The document discusses the use of Databricks as an analytics platform, highlighting features such as ETL workload management, various data types supported, and the integration of 3rd party data. It also emphasizes the importance of automation using tools like Airflow for workflow management, along with best practices for data handling and storage. Additionally, the document outlines YipitData's research offerings and encourages feedback and job applications.
Using Databricks as an Analysis Platform
- 2. Using databricks as an analysis platform Anup Segu
- 3. Agenda Extending databricks to provide a robust analytics platform Why a platform? What is in our platform?
- 4. Feedback Your feedback is important to us. Don’t forget to rate and review the sessions.
- 6. YipitData Answers Key Investor Questions ▪ 70+ research products covering U.S. and international companies ▪ Email reports, excel files, and data downloads ▪ Transaction data, web data, app data, targeted interviews, and adding more ▪ Clients include over 200 investment funds and fortune 500 companies ▪ 53 data analysts and 3 data engineers ▪ 22 engineers total ▪ We are rapidly growing and hiring!
- 7. About Me ▪ Senior Software Engineer ▪ Manage platform for ETL workloads ▪ Based out of NYC ▪ linkedin.com/in/anupsegu
- 8. We Want Analysts To Own The Product Data Collection Data Exploration ETL Workflows Report Generation www SELECT * FROM ...
- 11. Python Library Inside Notebooks
- 12. Ingesting data
- 13. Wide range of table sizes and schemas 1 PB Compressed Parquet 60 K Tables 1.7 K Databases
- 14. Readypipe: From URLs To Parquet ▪ Repeatedly capture a snapshot of the website ▪ Websites frequently change ▪ Makes data available quickly for analysis
- 15. Glue Metastore Streaming As JSON Is Great ▪ Append only data in S3 ▪ We don’t know the schema ahead of time ▪ Only flat column types s3://{json_bucket}/{project_name} /{table}/... JSON Bucket Parquet Bucket Kinesis Firehose
- 16. Parquet Makes Data “Queryable” ▪ Create or update databases, tables, and schemas as needed ▪ Partitioned by the date of ingestion ▪ Spark cluster subscribed to write events s3://{parquet_bucket}/{project_name} /{table}/dt={date}... JSON Bucket Kinesis Firehose Glue Metastore Parquet Bucket
- 17. Compaction = Greater Performance ▪ Insert files into new S3 locations ▪ Update partitions in Glue ▪ Pick appropriate column lengths for optimal file counts s3://{parquet_bucket}/{project_name} /{table}/compacted/dt={date}... JSON Bucket Kinesis Firehose Glue Metastore Parquet Bucket
- 18. With 3rd Party Data, We Strive for Uniformity Various File Formats Permissions Challenges Data Lineage Data Refreshes 403 Access Denied
- 19. Databricks Helps Manage 3rd Party Data ▪ Upload files and convert to parquet with additional metadata ▪ Configure data access by assuming IAM roles within notebooks
- 20. Table Utilities
- 21. Table: Database + Name + Data
- 23. Table Hygiene Pays Off Validate table naming conventions Keep storage layer organized Maintain prior versions of tables Automate table maintenance
- 24. However, Our Team Is Focused On Analysis so best practices are built into “create_table”
- 26. Wide Range Of Options For Spark Clusters Hardware Permissions Spark Configuration Driver instance Metastore Runtime Worker instances S3 access Spark properties EBS Volumes IAM Roles Environment Variables
- 27. Wide Range Of Options For Spark Clusters Hardware Permissions Spark Configuration Driver instance Metastore Runtime Worker instances S3 access Spark properties EBS Volumes IAM Roles Environment Variables
- 28. T-Shirt Sizes For Clusters ▪ 3 r5.xlarge instances ▪ Warm instance pool for fast starts ▪ 10 r5.xlarge instances ▪ Larger EBS volumes available if needed “MEDIUM”“SMALL” ▪ 30 r5.xlarge instances Larger EBS volumes for heavy workloads “LARGE” Standard IAM Roles, Metastore, S3 access, and Environment Variables
- 29. Launch Spark Jobs With Ease
- 30. Names Map To Databricks Configurations
- 31. Databricks Does The Heavy Lifting ▪ Provisions compute resources via a REST API ▪ Scales instances for cluster load ▪ Applies a wide range of spark optimizations
- 33. Airflow Is Our Preferred ETL Tool
- 34. Airflow Is Our Preferred ETL Tool Requires someone to manage this code
- 35. We use the databricks API to construct DAGs programmatically +
- 36. 1 DAG = 1 Folder, 1 Task = 1 Notebook
- 37. Templated Notebooks For DAGs /folder - commands - notebook_a - notebook_b - notebook_c
- 38. Translate Notebooks Into DAG files /api/2.0/workspace/list /api/2.0/workspace/export
- 39. Automatically Create Workflows ▪ Pipelines are deployed without engineers ▪ Robust logging and error handling ▪ Easy to modify DAGs ▪ All happens within databricks Task A Task B Task C
- 42. Standardize Logs As Data
- 43. Visualize Logs In Notebooks
- 44. A Platform Invites New Solutions ▪ Establish standard queries and notebooks ▪ Trigger one DAG from one another ▪ Trigger reporting processes after ETL jobs
- 45. Thank You Interested in working with us? We are hiring! yipitdata.com/careers
- 46. Feedback Your feedback is important to us. Don’t forget to rate and review the sessions.
- 47. Appendix I: Compaction Code
- 51. Appendix II: Table Creation Code
- 52. Capturing metadata with source data
- 53. Creating a table
- 54. Creating a table (cont.)
- 55. Creating a table (cont.)
- 56. Creating a table (cont.)
- 57. Appendix III: Databricks Jobs Code
- 58. Create a Databricks Job
- 59. Create a Databricks Job (cont.)
- 60. Appendix IV: Airflow Code
- 62. Automatic DAG Creation (cont.)
- 63. Automatic DAG Creation (cont.)
- 64. Automatic DAG Creation (cont.)