Bi and AI updates in the Microsoft Data Platform stack
0 likes671 views
This document provides an overview and agenda for a presentation on new features in the Microsoft data platform universe including SQL Server, Azure, and Power BI, with a focus on business intelligence (BI) and artificial intelligence (AI). The presentation covers new features in SQL Server 2019 for BI like calculation groups and dynamic formatting in Analysis Services tabular mode. It also highlights important updates in Azure including new serverless and hyperscale options for Azure SQL Database, updates to data ingestion and storage with Azure Data Factory v2 and Azure Data Lake Gen 2, and changes to data modeling, preparation, and serving capabilities in Azure SQL Data Warehouse. The document concludes by discussing options for incorporating AI and machine learning into BI solutions using either
Bi and AI updates in the Microsoft Data Platform stack
- 2. Powered by BI and AI in the Microsoft data platform universe (with a dash of cloud) Ivan Donev
- 3. Agenda • What’s new in SQL 2019 for BI • Important updates on Azure for Data platform • AI and ML for the masses
- 4. What is new in 2019 for BI • SQL Server AS • MD nothing • Tabular – Calc groups, M:M relationships, dynamic formatting • SQL Server IS • Nothing • SQL Server RS • Nothing • SQL Server MDM and DQS • Almost nothing
- 5. DEMO with AS 2019 Calculation groups Dynamic formatting M:M in tabular
- 7. Updates in Azure SQL DB • Azure SQL DB – Hyperscale• Azure SQL DB – Serverless
- 8. Azure SQL DB – Serverless • Single DB Serverless compute tier • Billed on compute used per SECOND • Used only in the vCore model • Parametrize the min/max vCores • Scenarios • Intermittent usage • Frequently rescaled DBs • New deployments prior historical usage data
- 9. Azure SQL DB – Hyperscale • Up to 100TB • Fast backups (filesystem snapshots) • Up to a minute restores • Faster throughput • Fast scale out and scale up • Distributed architecture
- 10. Updates in the DWH and Big Data world
- 11. The modern datawarehouse layout
- 12. Modern DWH – important updates in Ingest • Azure Data Factory v2 • Integration runtimes to run SSIS as-is • Storing SSIS catalogue in SQL DB • Mapping workflow • Wrangling workflow
- 13. Data factory v2 – Mapping workflow
- 14. Data factory v2 – wrangling dataflow
- 15. Modern DWH – important updates in Store • Azure Data Lake Gen 2 • Hierarchical file system • Security • Performance • Much easier to integrate with other services
- 16. Modern DWH – important updates in Prep and Train • Azure Databricks Delta • Spark engine with RDBMS features
- 17. Databricks Delta • ACID transactions • Versioned PARQUET files • Streaming writes to a table (i.e. Kafka) • Batch upserts • High performance reads • Schema enforcement
- 18. Modern DWH – important updates in Model and Serve • Changes in Azure DWH • Concurrency increased to 128 • Adaptive caching (NVMe !!!) • Unlimited Columnstore storage capacity • Workload classification and importance improvements • Changes in PowerBI
- 19. PowerBI Updates worth noting • PowerBI Dataflows • Self-service data transformation • Shared and certified datasets (preview) • Paginated Reports (SSRS) • Premium • XMLA Endpoints • Premium • Auto ML • Premium • CDS integration
- 20. The AI in BI • Options to use in DWH and BI
- 22. The AI in BI • Options to use in DWH and BI
- 23. The ML in BI Not scalableSelf-service AI •Prototyping •Do not need additional configuration or tuning •Options are •Microsoft Cognitive services with PowerBI (demo) •AutoML in Dataflows in PowerBI Premium •R/Python visuals Scalable, configurable, needs specialized staffEnterprise AI •Mandatory to run ML and store it in the Store/Serve model •Options are •Databricks •Azure ML •R/Python in Dataflow as data sources
- 24. How to choose? • The aim? • Prototype/Test/Verify • Production/O16N • The knowledge • R/Python/Scala/Java/… • The task • Image processing/Text analytics/Prediction/Classification • The post-production support • Can you support the solution afterwards?
- 25. Demo • PowerBI Dataflows • Using External Cognitive Services APIs
- 26. THANK YOU All my demos will be described and uploaded on our blog: http://sqlmasteracademy.com/techblog/
Editor's Notes
- #9: Serverless - https://docs.microsoft.com/en-us/azure/sql-database/sql-database-serverless Scenarios well-suited for serverless compute Single databases with intermittent, unpredictable usage patterns interspersed with periods of inactivity and lower average compute utilization over time. Single databases in the provisioned compute tier that are frequently rescaled and customers who prefer to delegate compute rescaling to the service. New single databases without usage history where compute sizing is difficult or not possible to estimate prior to deployment in SQL Database. Hyperscale - https://docs.microsoft.com/en-us/azure/sql-database/sql-database-service-tier-hyperscale Support for up to 100 TB of database size Nearly instantaneous database backups (based on file snapshots stored in Azure Blob storage) regardless of size with no IO impact on Compute Fast database restores (based on file snapshots) in minutes rather than hours or days (not a size of data operation) Higher overall performance due to higher log throughput and faster transaction commit times regardless of data volumes Rapid scale out - you can provision one or more read-only nodes for offloading your read workload and for use as hot-standbys Rapid Scale up - you can, in constant time, scale up your compute resources to accommodate heavy workloads as and when needed, and then scale the compute resources back down when not needed.
- #13: What exactly is the Azure DWH landscape at its core. All other services like Logic apps, functions, etc. are auxiliary to the main concept
- #14: What exactly is the Azure DWH landscape at its core. All other services like Logic apps, functions, etc. are auxiliary to the main concept
- #17: What exactly is the Azure DWH landscape at its core. All other services like Logic apps, functions, etc. are auxiliary to the main concept
- #18: What exactly is the Azure DWH landscape at its core. All other services like Logic apps, functions, etc. are auxiliary to the main concept
- #20: https://azure.microsoft.com/en-us/blog/adaptive-caching-powers-azure-sql-data-warehouse-performance-gains/ https://docs.microsoft.com/en-us/azure/sql-data-warehouse/sql-data-warehouse-workload-classification https://docs.microsoft.com/en-us/azure/sql-data-warehouse/sql-data-warehouse-workload-importance