Optimized your sql server operation using big data echo system
Download as PPTX, PDF0 likes429 views
The document discusses optimizing SQL Server operations using a big data ecosystem. It introduces several open source projects - Puppet for configuration management, Grafana for real-time dashboards, and Logstash, Elasticsearch and Kibana (LEK Stack) for log collection, storage and visualization. The presentation also provides an overview of Agoda's large production environment and data team roles and responsibilities.
Optimized your sql server operation using big data echo system
- 1. All material herein © 2005 – 2013 Agoda group of companies. All Rights Reserved. AGODA® is a registered trademark of AGIP LLC, used under license by Agoda Company Pte. Ltd. Agoda is part of priceline.com (NASDAQ:PCLN) 1 Optimized your sql server operation using big data echo system
- 2. All material herein © 2005 – 2013 Agoda group of companies. All Rights Reserved. AGODA® is a registered trademark of AGIP LLC, used under license by Agoda Company Pte. Ltd. Agoda is part of priceline.com (NASDAQ:PCLN) 2 Agenda • Introduction • Big data– The bad the good & the ugly • Open source projects • Puppet • Grafana • Logstash, ElasticSearch and Kibana • Summary
- 3. All material herein © 2005 – 2013 Agoda group of companies. All Rights Reserved. AGODA® is a registered trademark of AGIP LLC, used under license by Agoda Company Pte. Ltd. Agoda is part of priceline.com (NASDAQ:PCLN) 3 Agoda.com • Agoda.com is one of the world`s leading online hotel booking platforms. • We work with hundreds of thousands of hotels worldwide to offer discounted hotel rates to customers.
- 4. All material herein © 2005 – 2013 Agoda group of companies. All Rights Reserved. AGODA® is a registered trademark of AGIP LLC, used under license by Agoda Company Pte. Ltd. Agoda is part of priceline.com (NASDAQ:PCLN) 4 Agoda Production environment • 600 instances • Support 7 data engines • Sql server • Cassandra • Verica • Cubes • Elastic search • Spark • Hadoop • 4 data centers
- 5. All material herein © 2005 – 2013 Agoda group of companies. All Rights Reserved. AGODA® is a registered trademark of AGIP LLC, used under license by Agoda Company Pte. Ltd. Agoda is part of priceline.com (NASDAQ:PCLN) 5 Agoda data team
- 6. All material herein © 2005 – 2013 Agoda group of companies. All Rights Reserved. AGODA® is a registered trademark of AGIP LLC, used under license by Agoda Company Pte. Ltd. Agoda is part of priceline.com (NASDAQ:PCLN) 6 Data Operation Roles & Responsibilities • Installations & upgrade • Maintenance • Monitoring - active , proactive • Performance Tuning • Deploy database changes
- 7. All material herein © 2005 – 2013 Agoda group of companies. All Rights Reserved. AGODA® is a registered trademark of AGIP LLC, used under license by Agoda Company Pte. Ltd. Agoda is part of priceline.com (NASDAQ:PCLN) 7 Stairway (pyramid) to heaven Hardware Operating system Data engine Application
- 8. All material herein © 2005 – 2013 Agoda group of companies. All Rights Reserved. AGODA® is a registered trademark of AGIP LLC, used under license by Agoda Company Pte. Ltd. Agoda is part of priceline.com (NASDAQ:PCLN) 8 The bad the good & the ugly
- 9. All material herein © 2005 – 2013 Agoda group of companies. All Rights Reserved. AGODA® is a registered trademark of AGIP LLC, used under license by Agoda Company Pte. Ltd. Agoda is part of priceline.com (NASDAQ:PCLN) 9 Puppet • Puppet is an open source data center automation and configuration management framework. • Scalable to thousands of nodes • Web-based dashboard for reporting • Hundreds of community developed modules • puppetlabs/sqlserver
- 10. All material herein © 2005 – 2013 Agoda group of companies. All Rights Reserved. AGODA® is a registered trademark of AGIP LLC, used under license by Agoda Company Pte. Ltd. Agoda is part of priceline.com (NASDAQ:PCLN) 10
- 11. All material herein © 2005 – 2013 Agoda group of companies. All Rights Reserved. AGODA® is a registered trademark of AGIP LLC, used under license by Agoda Company Pte. Ltd. Agoda is part of priceline.com (NASDAQ:PCLN) 11 Puppet – use cases • Install • Upgrade • Migration –Storage , server • Deployment lifecycle – safe & fast
- 12. All material herein © 2005 – 2013 Agoda group of companies. All Rights Reserved. AGODA® is a registered trademark of AGIP LLC, used under license by Agoda Company Pte. Ltd. Agoda is part of priceline.com (NASDAQ:PCLN) 12 No upload, No cry Establish a baseline Identify bottlenecks OptimizeMeasure performance Repeat (if desired)
- 13. All material herein © 2005 – 2013 Agoda group of companies. All Rights Reserved. AGODA® is a registered trademark of AGIP LLC, used under license by Agoda Company Pte. Ltd. Agoda is part of priceline.com (NASDAQ:PCLN) 13 Grafana • Grafana provides a powerful and elegant way to create, explore, and share dashboards, • Open source • Real time • Feature Highlights • Annotations • Template Variables • Rich Graphing • play.grafana.org
- 14. All material herein © 2005 – 2013 Agoda group of companies. All Rights Reserved. AGODA® is a registered trademark of AGIP LLC, used under license by Agoda Company Pte. Ltd. Agoda is part of priceline.com (NASDAQ:PCLN) 14 Grafana architecture
- 15. All material herein © 2005 – 2013 Agoda group of companies. All Rights Reserved. AGODA® is a registered trademark of AGIP LLC, used under license by Agoda Company Pte. Ltd. Agoda is part of priceline.com (NASDAQ:PCLN) 15 Collect Source • Performance monitor • Query • XE • Error logs • JVM Method Unix – collectD Windows - Graphite-PowerShell Architecture Policy based Central collector vs local collector
- 16. All material herein © 2005 – 2013 Agoda group of companies. All Rights Reserved. AGODA® is a registered trademark of AGIP LLC, used under license by Agoda Company Pte. Ltd. Agoda is part of priceline.com (NASDAQ:PCLN) 16 Cluster management
- 17. All material herein © 2005 – 2013 Agoda group of companies. All Rights Reserved. AGODA® is a registered trademark of AGIP LLC, used under license by Agoda Company Pte. Ltd. Agoda is part of priceline.com (NASDAQ:PCLN) 17 Scale using replication Master server DC 1 DC 2 DC 3 DC 4 20xfront end 2xRPL 2xRPL 2xRPL 2xRPL 20x front end 12x front end 12x front end
- 18. All material herein © 2005 – 2013 Agoda group of companies. All Rights Reserved. AGODA® is a registered trademark of AGIP LLC, used under license by Agoda Company Pte. Ltd. Agoda is part of priceline.com (NASDAQ:PCLN) 18 Monitor Replication
- 19. All material herein © 2005 – 2013 Agoda group of companies. All Rights Reserved. AGODA® is a registered trademark of AGIP LLC, used under license by Agoda Company Pte. Ltd. Agoda is part of priceline.com (NASDAQ:PCLN) 19 Replication – performance monitor counters Agent Performance object Counter Description Log Reader Agent SQL Server: Replication Logreader Logreader: Delivered Cmds/sec The number of commands per second delivered to the Distributor. Log Reader Agent SQL Server: Replication Logreader Logreader: Delivered Trans/sec The number of transactions per second delivered to the Distributor. Log Reader Agent SQL Server: Replication Logreader Logreader: Delivery Latency The current amount of time, in milliseconds, elapsed from when transactions are applied at the Publisher to when they are delivered to the Distributor. Distribution Agent SQL Server: Replication Dist. Dist: Delivered Cmds/sec The number of commands per second delivered to the Subscriber. Distribution Agent SQL Server: Replication Dist. Dist: Delivered Trans/sec The number of transactions per second delivered to the Subscriber. Distribution Agent SQL Server: Replication Dist. Dist: Delivery Latency The current amount of time, in milliseconds, elapsed from when transactions are delivered to the Distributor to when they are applied at the Subscriber.
- 20. All material herein © 2005 – 2013 Agoda group of companies. All Rights Reserved. AGODA® is a registered trademark of AGIP LLC, used under license by Agoda Company Pte. Ltd. Agoda is part of priceline.com (NASDAQ:PCLN) 20 Logstash, ElasticSearch & Kibana Logstash • Centralize data processing of all types • Normalize varying schema and formats • Quickly extend to custom log formats • ElasticSearch • Search server based on Lucene. • Scaleable • Kibana • Flexible analytics and visualization platform • Real-time summary and charting of streaming data
- 21. All material herein © 2005 – 2013 Agoda group of companies. All Rights Reserved. AGODA® is a registered trademark of AGIP LLC, used under license by Agoda Company Pte. Ltd. Agoda is part of priceline.com (NASDAQ:PCLN) 21 Logstash, ElasticSearch & Kibana
- 22. All material herein © 2005 – 2013 Agoda group of companies. All Rights Reserved. AGODA® is a registered trademark of AGIP LLC, used under license by Agoda Company Pte. Ltd. Agoda is part of priceline.com (NASDAQ:PCLN) 22 Elastic Search – Use cases • Application logs • Data engine logs • OS logs • Hardware logs - network , storage • Free text search
- 23. All material herein © 2005 – 2013 Agoda group of companies. All Rights Reserved. AGODA® is a registered trademark of AGIP LLC, used under license by Agoda Company Pte. Ltd. Agoda is part of priceline.com (NASDAQ:PCLN) 23 Kapunkap
- 24. All material herein © 2005 – 2013 Agoda group of companies. All Rights Reserved. AGODA® is a registered trademark of AGIP LLC, used under license by Agoda Company Pte. Ltd. Agoda is part of priceline.com (NASDAQ:PCLN) 24 We are looking for new talents! • Are you ready to work in a fast paced environment, with opportunity for career progression? • Do you want to be part of a multicultural team, with colleagues from all around the world? • Do you want to utilize and improve your foreign language skills further? • Are you ready to take up challenges and give prompt solutions for customers with their requests?
Editor's Notes
- #14: http://play.grafana.org/dashboard/db/table-panel-showcase