Managing and querying large data sets using Data Factory, Cosmos DB and Azure Functions
2 likes691 views
The document outlines a case related to a murder in New York City on January 29, 2014, focusing on identifying which taxi the suspect used. It discusses the use of various datasets, including NYPD complaint data and NYC taxi trip data, and highlights the capabilities of Azure services like Data Factory, Cosmos DB, and Azure Functions for managing and querying these datasets. Additionally, it includes references to hands-on labs for practical application of the discussed technologies.
![Think ahead. Act now.
Cosmos DB: GeoJSON
https://docs.microsoft.com/en-us/azure/cosmos-db/geospatial
Azure Cosmos DB supports indexing and querying of
geospatial point data that's represented using
the GeoJSON specification.
{
"type":"Point",
"coordinates":[-73.88, 40.76]
}](https://image.slidesharecdn.com/gabc2018xpiritmanagingandqueryingdatasets-180421112734/85/Managing-and-querying-large-data-sets-using-Data-Factory-Cosmos-DB-and-Azure-Functions-11-320.jpg)
Managing and querying large data sets using Data Factory, Cosmos DB and Azure Functions
- 1. Think ahead. Act now. Think ahead. Act now. Managing and querying datasets with Data Factory, Cosmos DB and Azure Functions. Marc Duiker
- 2. Think ahead. Act now. The Case A murder has happened in New York City on 29th Jan 2014. The suspect most likely escaped by using a taxi. It’s our job to find out which taxi the suspect could have used.
- 3. Think ahead. Act now. Original data sources NYPD Complaint Data 2014 NYC Taxi Trip Data 2014 https://data.cityofnewyork.us/Public-Safety/NYPD- Complaint-Data-Historic/qgea-i56i https://www.kaggle.com/kentonnlp/2014-new-york- city-taxi-trips 5.6M rows 1.3 GB 15M rows 2.3 GB
- 4. Think ahead. Act now. NYPD Complaint Data (trimmed down) CSV with 39k records for Jan 2014 Attributes • Date & time • Offense classification (KY_CD) • Latitude & longitude • … More details in NYPD_Complaint_Data_Column_Descriptions.csv
- 5. Think ahead. Act now.
- 6. Think ahead. Act now. NYC Taxi Trip Data (trimmed down) CSV with 477k records for 29th Jan 2014 Attributes • Pickup date & time • Pickup latitude & longitude • …
- 7. Think ahead. Act now.
- 8. Think ahead. Act now. What can we use on Azure? Query Azure FunctionsData FactoryCosmos DB Storage TransferCsv
- 9. Think ahead. Act now. Cosmos DB
- 10. Think ahead. Act now. Cosmos DB: databases & collections Account (gabc-nyc-db) Database (nycdatabase) Collection (complaints) Collection (taxitrips) Documents { .. } Documents { .. } SQL API
- 11. Think ahead. Act now. Cosmos DB: GeoJSON https://docs.microsoft.com/en-us/azure/cosmos-db/geospatial Azure Cosmos DB supports indexing and querying of geospatial point data that's represented using the GeoJSON specification. { "type":"Point", "coordinates":[-73.88, 40.76] }
- 12. Think ahead. Act now. Cosmos DB: Change feed • Cosmos DB persists events about insertion and updates to documents in the change feed.
- 13. Think ahead. Act now. Cosmos DB
- 14. Think ahead. Act now. Azure Data Factory (v2) Source Sink • Mapping • Transforms • Scheduling • Throughput
- 15. Think ahead. Act now. Data Factory: source schema • When importing a csv DataFactory looks at the first line of data to determine the data types. • Inspect the data for empty and numeric values • “” empty value for String, Int64 or Double? • 0 Int64 or Double? • Sometimes numbers are categories (String).
- 16. Think ahead. Act now. Azure Data Factory https://datafactoryv2.azure.com/
- 17. Think ahead. Act now. Azure Functions (Runtime 2) • Serverless compute service to run code on demand • Support for various languages: C#, F#, Node.js, Java, or PHP • Automatic scaling • Pay-per-use
- 18. Think ahead. Act now. Azure Functions Triggers
- 19. Think ahead. Act now. Azure Functions
- 20. Think ahead. Act now. Integrating the services UpdateTaxiTripGeoData NYPD Complaint CSV NYC Taxi Trip CSV Data Factory Pipelines Cosmos DB Collections GetComplaintsByOffenseId GetTaxiTripsWithinRange Lab 1 Lab 2
- 21. Think ahead. Act now. Hands-on labs https://github.com/XpiritBV/GABC2018_HandsOnLabs/