Lunch and Learn: You have the data, now what?
0 likes29 views
The document provides an agenda and overview for a "Lunch & Learn" event on data and analytics at DiUS. It introduces DiUS as a company specializing in emerging technologies and discusses some common data terms. It then outlines the agenda which includes an introduction, demystifying some data terms, aspects of data engineering vision and database selection, and a panel discussion. Examples of DiUS's work with two clients focusing on taking a product lens to data problems are also summarized.
Lunch and Learn: You have the data, now what?
- 1. 2022 1
- 2. 25th July 2023 2 Lunch & learn You have the data, now what?
- 3. Who is DiUS? ● Ideation, innovation and labs ● Discovery services ● Experience and services design ● Digital product development ● Application modernisation ● Data platforms, analytics and AI/machine learning ● Internet of Things Our services. 19 Years. 150 People. Australia + New Zealand Specialists in using emerging technology to solve difficult problems, get new ideas to market or disrupt traditional business models.
- 4. Got a question? slidio.com | #2040429
- 5. Data and analytics at DiUS What we’re doing/ what we’re seeing
- 6. 6 Meet your speakers Nabi Rezvani Lead Data Engineer E: [email protected] Gaurav Thadani Lead Software Engineer E: [email protected]
- 7. ● Introduction ● Problem space - What are we dealing with - Explosion of tools - The process forward ● Product led thinking - Applied lens - Themes - Approach ● Panel discussion Agenda
- 12. ● Data lake ● Data warehouse ● Data lakehouse ● Delta lake ● Data mesh ● Data fabric Some terms to demystify?
- 13. Aspects - DB Selection 13
- 14. Aspects - Data engineering vision 14
- 15. Lastly 15
- 16. Data as a product Product thinking towards data
- 17. Tech vs product lens Different lenses towards data initiatives Data architecture and infrastructure Data processing Data visualisation Data streaming Data pipelines Data governance ML engineering Technology lens Data led decision making Add business value using data products Measure success using data Predict patterns using data Product lens
- 18. Just like software products, we start from the business objective, then define the product specifications. We need to build an API that shows us the best selling products per category on a daily basis. “ ” Actor Functional requirements Non-functional requirements Build a popular products API Marketing manager Show best selling products per category Update daily We do not have visibility on the best selling products to run marketing campaigns on them. “ ” Increase revenue by upselling popular products via marketing campaigns How? What? Why? Product lens towards data
- 19. Building a little bit in each layer, not the entire layer The “thin slice” approach
- 21. DiUS did some data engineering work with “Organisation X”. The history: ● Organisation X, had invested in building their shiny data platform for three years. ● The objective was to build a self served, scalable and data platform which makes the data discoverable and allows performant data processing. Case study 1: Organisation X
- 22. We have a lot of data across our verticals, let’s get everyone to ingest their data into the platform! 22 Case study 1: Organisation X
- 23. The “Data warehouse integration” project: ● Building the platform did not start with asking “what business problem we do we want to address?” in the first place. ● The integration of the new data warehouse did not seem to add much value. A business led project: ● Eventually a product leader came onboard and started bringing business problems to the table to solve using the platform and the data(calculate the annual recurring revenue). Case study 1: Organisation X Recap of the Organisation X situation.
- 24. We gathered around the table with questions like: ● “Can we have more visibility on the installation performance of the devices to measure the sales metrics of the partners faster?” ● “Can we have the fleet view data of the battery and bluetooth performance to be able to make informed product roadmap decisions?” ● “Can we ingest the energy data and make it queribale so that we can build new features around energy consumption patterns?” Case study 2: Powersensor Powersensor is a self-installed energy measurement product that enables households to monitor their home energy consumption and solar generation easily.
- 25. 25 Case study 2: Powersensor
- 26. Case study 2: Powersensor
- 27. Recap of the Powersensor use case: ● We initially started with the problem that we wanted to solve and focused on the business outcome. ● We built thin vertical slices rather than making a huge investment in making the best “data platform”. ● We focused on the functional requirements and did a trade-off between non-functional requirements along the way to keep the solution cost low. Case study 2: Powersensor
- 28. You have the data, now what? Panel discussion
- 29. 29 Meet your speakers Jonathan Gomez Head of Data Platforms, Wesfarmers OneDigital - OnePass John Sullivan CEO, ChargeFox
- 30. Got a question? slidio.com | #2040429
- 31. Know what you need/want Avoid buying technology or using products just like that. Wire up to product requirements. ● Interviews and discussions with the product to understand the current status of the platform and the business ● Anticipated requirements - short-mid-long term goals ● Conversation on why analytics? ○ Who needs to view it? ● Product Demo and walkthrough ● Review of the current product vision and roadmap ● Review of current backlog and proposed MVP Presentation title 31
- 32. Trends in D&A 2023
- 33. Data & tech lens What are the data engineering tools and best practices in the market?
- 34. PRESALES FRAMEWORK Data product Collecting and making available data as it is (perhaps with some small processing or cleansing steps). User does all the processing. Doing some of the processing on our side. Example: with customer data, add additional attributes like assigning a customer segment to each customer. Receive some data, run it through the algorithm (machine learning or otherwise) then return information or insights. Example: Google Image (user uploads a picture, and receives a set of images that are the same or similar) Provide information to the user to help them with decision-making but we are not taking the decision ourselves. Examples: sales dashboard, Google analytics Outsource intelligence within a specific domain. Example: recommending products to users From a blog post by Simon O’Regan https://towardsdatascience.com/designing-data-products-b6b93edf3d23
- 35. PRESALES FRAMEWORK Data product From a blog post by Simon O’Regan https://towardsdatascience.com/designing-data-products-b6b93edf3d23 Voice, robotics, augmented reality
- 36. Chatbots segmentation dashboards ML models Recommendation systems Data product matrix 36
- 37. From a blog post by Simon O’Regan https://towardsdatascience.com/designing-data-products-b6b93edf3d23 Chatbots segmentation dashboards ML models Recommendation systems Data product matrix 37
- 38. Technology lens ○ Design, build and revamp data platforms ○ Build data processing pipelines ○ Improve the performance of data processing pipelines Product lens ○ Address a business problem using data ○ Design data products ○ Build, ship and surface data products Data services Different lenses towards data initiatives 38
- 39. Technology lens Product lens Data services Different lenses towards data initiatives 39 Data architecture and infrastructure Data processing Data visualisation Data streaming Data pipelines Data governance ML engineering Data led decision making Add business value using data products Measure success using data Predict patterns using data
- 40. Just like software world, we could approach each data opportunity with the product lens as well. With software products, we start from the business objective, then define use cases and actors and specify functional and non-functional requirements. Product lens towards data We need to build a sales dashboard that shows us the best selling products per category in a real time manner. “ ” Use case Actor Functional requirements Non-functional requirements Build a sales dashboard Marketing manager Show best selling products per category Real-time (10 seconds latency) We do not have visibility on the best selling products to recommend them to more users. “ ” 40 Business objective Upselling the best selling products
- 41. They ended up with a complicated ecosystem of components, a lot of complexity and cost to deal with. Motivation of the project: ● We have 1600 datasets in the platform and zero users, let’s change that by making people’s lives easier. How about buying “Data warehouse Y”? Outcome of the project: ● “Data warehouse Y” was integrated, but there was still no interest in the platform! 41 Use case 1
- 42. The Data warehouse integration project: ● Building the platform did not start with asking “what business problem we do we want to address?” in the first place. ● No functional requirements and business outcomes were discussed in the first place. ● The integration of the new data warehouse did not seem to add much value A business led project: ● Eventually a product leader came onboard and started bringing business problems to the table to solve using the data and the platform (calculate the annual recurring revenue). Recap of the Organisation X situation. Use case 1
- 43. Powersensor is a self-installed energy measurement product that enables households to monitor their home energy consumption and solar generation easily. We gathered around the table with questions like: ● “Can we have more visibility on the onboarding of the devices to improve customer retention?” ● “Can we have the fleet view data of the battery and bluetooth performance to be able to make informed product roadmap decisions?” ● “Can we ingest the energy data and make it queribale for further exploration?” Case study 2:Powersensor
- 44. 44 Case study 2:Powersensor Research Build Rollout Plan Backlog of tasks Prioritise into phases 1,2,.. Define non-functional requirements Cost analysis Data readiness assessment Build & test Ship to prod Data onboarding Iterate for each phase...
- 45. cost latency The devices are sending data constantly, can we build a real time dashboard showing battery and bluetooth signal status? Data engineer Product owner The solution looks really good but I can see that there is a spike in the AWS cost Yes we can, let’s design and test a streaming solution to achieve that Alright, let’s go with a batch design then We can live with a few hours latency, keeping the cost down is very important though. Compromising the latency for lower cost Yeah, streaming solutions are expensive in general, unless we don’t need to build a real time dashboard. Case study 2:Powersensor
- 46. Case study 2: Powersensor