Using C# with U-SQL (SQLBits 2016)
Download as pptx, pdf0 likes794 views
The document outlines the key features and functionalities of U-SQL, a data processing language developed by Microsoft that combines the capabilities of C# with SQL-like syntax. It discusses user-defined functions, aggregation, and data extraction methods, as well as assembly management for building scalable data applications. Additional resources and examples for further exploration of U-SQL are also provided.
![Managing
Assemblies
Create assemblies
Reference assemblies
Enumerate assemblies
Drop assemblies
• CREATE ASSEMBLY db.assembly FROM @path;
• CREATE ASSEMBLY db.assembly FROM byte[];
• Can also include additional resource files
• REFERENCE ASSEMBLY db.assembly;
• Referencing .Net Framework Assemblies
• Always accessible system namespaces:
• U-SQL specific (e.g., for SQL.MAP)
• All provided by system.dll system.core.dll
system.data.dll, System.Runtime.Serialization.dll,
mscorelib.dll (e.g., System.Text,
System.Text.RegularExpressions, System.Linq)
• Add all other .Net Framework Assemblies with:
REFERENCE SYSTEM ASSEMBLY [System.XML];
• Enumerating Assemblies
• Powershell command
• U-SQL Studio Server Explorer
• DROP ASSEMBLY db.assembly;](https://image.slidesharecdn.com/3-sqlbits-usqlcsharpintegration-160505123558/85/Using-C-with-U-SQL-SQLBits-2016-6-320.jpg)
Using C# with U-SQL (SQLBits 2016)
- 1. Michael Rys Principal Program Manager, Big Data @ Microsoft @MikeDoesBigData, {mrys, usql}@microsoft.com Using C# with U-SQL 2016/04/04
- 2. Extensible From Ground Up • Type system is based on C# • Expression language IS C# • User-defined functions (U-SQL and C#) • User-defined Aggregators (C#) • User-defined Operators (UDO) (C#) U-SQL provides the Parallelization and Scale-out Framework for Usercode • EXTRACTOR, OUTPUTTER, PROCESSOR, REDUCER, COMBINER, APPLIER REFERENCE MyDB.MyAssembly; CREATE TABLE T( cid int, first_order DateTime , last_order DateTime, order_count int , order_amount float ); @o = EXTRACT oid int, cid int, odate DateTime, amount float FROM "/input/orders.txt" USING Extractors.Csv(); @c = EXTRACT cid int, name string, city string FROM "/input/customers.txt" USING Extractors.Csv(); @j = SELECT c.cid, MIN(o.odate) AS firstorder , MAX(o.date) AS lastorder, COUNT(o.oid) AS ordercnt , AGG<MyAgg.MySum>(c.amount) AS totalamount FROM @c AS c LEFT OUTER JOIN @o AS o ON c.cid == o.cid WHERE c.city.StartsWith("New") && MyNamespace.MyClass.MyFunction(o.odate) > 10 GROUP BY c.cid; OUTPUT @j TO "/output/result.txt" USING new MyData.Write();
- 6. Managing Assemblies Create assemblies Reference assemblies Enumerate assemblies Drop assemblies • CREATE ASSEMBLY db.assembly FROM @path; • CREATE ASSEMBLY db.assembly FROM byte[]; • Can also include additional resource files • REFERENCE ASSEMBLY db.assembly; • Referencing .Net Framework Assemblies • Always accessible system namespaces: • U-SQL specific (e.g., for SQL.MAP) • All provided by system.dll system.core.dll system.data.dll, System.Runtime.Serialization.dll, mscorelib.dll (e.g., System.Text, System.Text.RegularExpressions, System.Linq) • Add all other .Net Framework Assemblies with: REFERENCE SYSTEM ASSEMBLY [System.XML]; • Enumerating Assemblies • Powershell command • U-SQL Studio Server Explorer • DROP ASSEMBLY db.assembly;
- 7. Assembly Dependencies • Assembly must be registered to be referenced • All Assemblies needed for compilation must be referenced in script • All Assemblies needed at runtime either • Need to be referenced in script, or • Need to be registered with the assembly as additional files • Metadata Service does NOT enforce dependencies • Visual Studio Extension provides support for dependency management
- 8. Additional Resources MSDN Article https://msdn.microsoft.com/en- us/magazine/mt614251 Sample Data https://github.com/Azure/usql/tree/master/Exampl es/Samples/Data/Tweets Sample Project https://github.com/Azure/usql/tree/master/Exampl es/TweetAnalysis