ADL/U-SQL Introduction (SQLBits 2016)

Editor's Notes

• #4: The Data Warehouses leverages the top-down approach where there is a well-architected information store and enterprisewide BI solution. To build a data warehouse follows the top-down approach where the company’s corporate strategy is defined first. This is followed by gathering of business and technical requirements for the warehouse. The data warehouse is then implemented by dimension modelling and ETL design followed by the actual development of the warehouse. This is all done prior to any data being collected. It utilizes a rigorous and formalized methodology because a true enterprise data warehouse supports many users/applications within an organization to make better decisions.
• #5: A data lake is an enterprise wide repository of every type of data collected in a single place. Data of all types can be arbitrarily stored in the data lake prior to any formal definition of requirements or schema for the purposes of operational and exploratory analytics. Advanced analytics can be done using Hadoop, Machine Learning tools, or act as a lower cost data preparation location prior to moving curated data into a data warehouse. In these cases, customers would load data into the data lake prior to defining any transformation logic. This is bottom up because data is collected first and the data itself gives you the insight and helps derive conclusions or predictive models.
• #17: Other points to make here, but not called out above Built on Apache YARN Scales dynamically with the turn of a dial Supports Azure AD for access control, roles, and integration with on-prem identity systems U-SQL’s scalable runtime processes data across multiple Azure data sources
• #19: DATA SOURCE: Represents a remote data source such as Azure SQL Database. Have to specify all the details (connection string, credentials, etc required to connect to and issues queries. EXTERNAL TABLE: A local table, with columns defined in C# types, that redirects queries issued against it to the remote table that it is based on. U-SQL automatically does the type conversion. External tables lets you impose a specific schema against the remote data, shielding you from remote schema changes. You can issue queries that ‘join’ external and local tables. PASS THROUGH queries: These queries are issued directly against the remote data source in the syntax of the remote data source (say T-SQL for Azure SQL database). REMOTABLE_TYPES: For every external data source you have to specify the list of ‘remoteable types. This list constrains the types of queries that will be remoted. Ex: REMOTABLE_TYPES = (bool, byte, short, ushort, int, decimal); LAZY METADATA LOADING: Here the remote data schematized only when the query is actually issues to the remote data source. Your program must be able to deal with remote schema changes.
• #22: Add velocity?
• #23: Hard to operate on unstructured data: Even Hive requires meta data to be created to operate on unstructured data. Adding Custom Java functions, aggregators and SerDes is involving a lot of steps and often access to server’s head node and differs based on type of operation. Requires many tools and steps. Some examples: Hive UDAgg Code and compile .java into .jar Extend AbstractGenericUDAFResolver class: Does type checking, argument checking and overloading Extend GenericUDAFEvaluator class: implements logic in 8 methods. - Deploy: Deploy jar into class path on server Edit FunctionRegistry.java to register as built-in Update the content of show functions with ant Hive UDF (as of v0.13) Code Load JAR into head node or at URI CREATE FUNCTION USING JAR to register and load jar into classpath for every function (instead of registering jar and just use the functions)
• #24: Spark supports Custom “inputters and outputters” for defining custom RDDs No UDAGGs Simple integration of UDFs but only for duration of program. No reuse/sharing. Cloud dataflow? Requires has to care about scale and perf Spark UDAgg Is not yet supported ( SPARK-3947) Spark UDF Write inline functiondef westernState(state: String) = Seq("CA", "OR", "WA", "AK").contains(state) for SQL usage need to register the tablecustomerTable.registerTempTable("customerTable") Register each UDFsqlContext.udf.register("westernState", westernState _) Call itval westernStates =sqlContext.sql("SELECT * FROM customerTable WHERE westernState(state)")
• #25: Offers Auto-scaling and performance Operates on unstructured data without tables needed Easy to extend declaratively with custom code: consistent model for UDO, UDF and UDAgg. Easy to query remote sources even without external tables U-SQL UDAgg Code and compile .cs file: Implement IAggregate’s 3 methods :Init(), Accumulate(), Terminate() C# takes case of type checking, generics etc. Deploy: Tooling: one click registration in user db of assembly By Hand: Copy file to ADL CREATE ASSEMBLY to register assembly Use via AGG<MyNamespace.MyAggregate<T>>(a) U-SQL UDF Code in C#, register assembly once, call by C# name.
• #26: Remove SCOPE for external customers?
• #27: C# is the extension story for U-SQL Expressions in SELECT statement User-defined operators (UDOs) User-defined functions (UDFs) User-defined aggregates (UDAGGs) User-defined types (UDTs) UDOs are central to U-SQL user experience UDFs, UDAGGs, UDOs and UDTs require assemblies to be registered (one-time cost, fixed assembly version) UDFs UDAGGs, UDOs and UDTs will automatically be available after referencing assembly in script One version of assembly per database Assembly with same short name is not allowed Tooling provides code-behind and aut-odeploy experience
• #29: Use for language experts
• #32: Additional resources Blogs and community page: http://usql.io http://blogs.msdn.com/b/visualstudio/ http://azure.microsoft.com/en-us/blog/topics/big-data/ https://channel9.msdn.com/Search?term=U-SQL#ch9Search Documentation: http://aka.ms/usql_reference https://azure.microsoft.com/en-us/documentation/services/data-lake-analytics/ ADL forums and feedback http://aka.ms/adlfeedback https://social.msdn.microsoft.com/Forums/azure/en-US/home?forum=AzureDataLake http://stackoverflow.com/questions/tagged/u-sql