Adding Open Data Value to 'Closed Data' Problems
Download as PPTX, PDF1 like668 views
The document discusses the integration of open data with closed data to enhance data science applications, particularly in the context of conference paper review processes and fraud detection. It highlights the role of open research data and its applications across various sectors, emphasizing the value derived from combining different data sources. Additionally, it outlines practical implementations such as the use of subsift for expert finding and assurance scoring in data analytics.
Adding Open Data Value to 'Closed Data' Problems
- 1. Adding Open Data Value to 'Closed Data' Problems Dr Simon Price Research Fellow, University of Bristol Data Scientist, Capgemini Insights & Data
- 2. Who am I? • 30 years software development and leadership roles • Moved into Data Science via PhD in Machine Learning (2014) • Research Fellow in Machine Learning group ~20 Machine Learning researchers • Led project to establish Bristol’s open research data repository • One of the organisers of Open Data Institute (ODI) Bristol • Data Scientist in Big Data Analytics team ~100 Data Scientists, Big Data Engineers and Data Analysts • Focus on Open Source and Big Data technologies to solve client problems
- 3. Outline 1. Case study: open data + ‘closed data’ 2. Deriving value from open data 3. Data Science with ‘closed data’
- 4. Case study: SubSift Conferences using SubSift • ECML-PKDD: European Conference on Machine Learning and Principles and Practice of Knowledge Discovery in Databases • KDD: ACM SIGKDD International Conference on Knowledge Discovery and Data Mining • PAKDD: Pacific-Asia Conference on Knowledge Discovery and Data Mining • SDM: SIAM International Conference on Data Mining Journals using SubSift • Machine Learning • Data Mining and Knowledge Discovery https://doi.org/10.1145/2979672
- 5. Initial problem addressed by SubSift Matching submitted conference papers to possible reviewers in Programme Committee
- 10. Expert finding
- 11. Why did SubSift recommend this person?
- 13. Profiling staff at meetings
- 15. Open research data • data.bris.ac.uk • Research data storage facility • Each researcher gets 10TB "forever"
- 17. 140+ datasets live on opendata.bristol.gov.uk Mostly static but some real-time data Examples • Government: Elections since 2007 • Community: Quality of Life survey • Education: School Results • Energy: Installed PV, Energy Use in Council Buildings • Environment: Real time & Historic Air Quality, Flood Alerts (EA) • Land use: 2013 Planning applications • Health: Life expectancy/ Mortality, Obesity, NHS Spend Open government data
- 19. Deriving value from open data 1. Data Science 2. Using open data to enrich and connect ’closed data’
- 22. statistics software engineering machine learning data science application domains research domains
- 23. Big Data Analytics Insights & Data www.capgemini.com/insights-data
- 24. 25Copyright © Capgemini 2017. All Rights Reserved June 2017 Example Data Science application Assurance Scoring http://ow.ly/4nbEUI Using existing enterprise data plus any useful open data, detect potentially fraudulent transactions
- 25. 26Copyright © Capgemini 2017. All Rights Reserved June 2017 Example Data Science application Assurance Scoring http://ow.ly/4nbEUI
- 26. 27Copyright © Capgemini 2017. All Rights Reserved June 2017 Machine Learning Transform Selection Model Training Validation Test Feature Extraction and Selection Model Building Variety of output files: logs, graphics, saved models, etc. Testing: Unit tests, monitoring tests and integration tests Vector Build Input Data Manipulate, Explore Data Machine Learning Framework (Python, Scala, Spark)
- 27. 28Copyright © Capgemini 2017. All Rights Reserved June 2017 Graph Links - Matching Key part of assurance scoring – bringing data together from disparate sources Probability of Match: 80% Attribute Data Source 1 Data Source 2 Name Richard Smith Rich Smith Phone Number 07123 456 789 07123 456 798 Favourite Sport Football Cricket
- 28. 29Copyright © Capgemini 2017. All Rights Reserved June 2017 Related to: - record linkage - duplicate detection - reference resolution - object identity - entity matching Connect graph descriptions using background knowledge from open data sources. e.g. Linked Open Data Advanced matching
- 29. 30Copyright © Capgemini 2017. All Rights Reserved June 2017 Linked Open Data
- 30. Data Science with ‘closed data’ The information contained in this presentation is proprietary. © 2012 Capgemini. All rights reserved. www.capgemini.com About Capgemini With more than 120,000 people in 40 countries, Capgemini is one of the world's foremost providers of consulting, technology and outsourcing services. The Group reported 2011 global revenues of EUR 9.7 billion. Together with its clients, Capgemini creates and delivers business and technology solutions that fit their needs and drive the results they want. A deeply multicultural organization, Capgemini has developed its own way of working, the Collaborative Business ExperienceTM, and draws on Rightshore ®, its worldwide delivery model. Rightshore® is a trademark belonging to Capgemini
- 31. Problems of opening up ‘closed data’
- 32. Research data now open by default - including sensitive data Funders Journals data.bris has 3 levels of access:
- 37. Data Science with ‘closed data’
- 38. Data science with ‘closed data’ • Custom R server running inside secure data repository / warehouse • Enables non-disclosive, remote analysis of sensitive research data.
- 39. Number of Letters NumberofWords Non-disclosive Disclosive
- 43. DataSHIELD partition models horizontal verticalideal