Python vs JLizard.... a python logging experience
1 like423 views
The document discusses log management as a service, emphasizing the exponential growth of log data and its challenges for organizations. It compares Python and Jlizard in the context of log management solutions, highlighting features of the Logentries agent and its capabilities. The need for effective logging is underscored by compliance requirements and the consequences of inadequate log management.
Python vs JLizard.... a python logging experience
- 1. Log Management As A Service Oct 2011 Python vs. JLizard
- 2. Outline • A Logging Story • About Us • Data Data Everywhere (i.e. Logs Logs Everywhere) • LogEntries.com • Logging in the Cloud & PAAS • Python vs JLizard
- 4. Can you see the error? 25 log events in this window
- 5. Can you see the error? Consider 10,000 events per second
- 7. JLizard & Logentries • UCD spin out company • Founded by Viliam Holub & Trevor Parsons • Based on UCD/IBM Research (Enterprise Ireland) • Participated in National Digital Research Centre’s Launchpad program (styled on Ycombinator) • Nova UCD spin out award • EI Funded Company • Dogpatch Europe Company • Logentries Launched in Q3 2011
- 8. About Me • Trevor Parsons – Tech Background • Enterprise Software Design, Monitoring Tools, Data Mining • Java Java Java • Performance & Testing Tools – Startups • Crovan: Training & Consultancy • JLizard
- 9. Data Data Everywhere (i.e. Logs Logs Everywhere) • Estimates are that 150 billion gigabytes (exabytes) of data was created in 2005; This year about 8 times that amount (1,200 exabytes) will be created - The Economist Feb 2010 • “The amount of enterprise data will grow about 650% over the next five years, the vast majority of it unstructured, or not included in any database. “ - Gartner 2009 • Log data is the fasted-growing data source at large organizations - CNET magazine 2010 • Many organizations are currently producing terabytes of log data per month - CIO.Net 2009
- 10. Log Maths 100,000 log messages / second x 300 bytes / log message ~ 28.6 MB x 3600 seconds ~ 100.6 GB / hour x 24 hours ~ 2.35 TB / day x 365 days ~ 860.5 TB / year x 3 years ~ 2.52 PB From Anton Chuvakin’s Blog Aug 2010 http://chuvakin.blogspot.com/
- 11. Typical Log Volumes Customer Type Log Volumes Events per Second Events per Day Large Cloud Provider 50 Terabyes per Day 2,000,000 172,000,000,000 Large Social Media 25 Terabytes per Day 1,000,000 Organisation Telecom Middleware/ 1 Terabyte per Day 50,0000 Applications Large Organisation 300 GB Per Day 15,000 (>1000 employees) Online Marketing Org 100 GB per day 5,000 432,000,000 Small 10 GBs per Day 500 Data Centre SAAS Educational 5Gbs Per Day 250 Tools Single IBM Test Team 2 GBs per Day 100 Online Multimedia 700Mbs Per Day 35 Early Stage Start up 50Mbs Per Day 25 2,000,000
- 12. Consequences …Hacked System …Slow Web Site …System Crash “Every minute that Facebook is down costs Zynga $10,000” SocialMediaInfluence.com, September 2010
- 13. Logging - A Cloud Requirement Server to Admin Ratio Increasing Significantly, from 10:1 to 500:1 Computerworld.com, July 2010 Compliance and Security Legislation Driving Requirement for Log Management Where Is My Elastic Log Management Service? securecloudreview.com & networkworld, September 2010
- 14. PAAS Logging • Limited Logging Infrastructure – App Engine – limited log buffers – Heroku – 500 events for free • No/Limited access to File System • No Log Storage • How to debug without any logs????
- 15. Python vs. JLizard Python @logentries • Logentries Agent • Web interface – Django • App Engine Plug ins
- 16. Logentries Python Agent Why Python? • Light weight agent • Runs on Linux Out Of The Box • Windows support – created an exe
- 18. Logentries Python Agent Features: Available on github: - Authenticate with Logentries Account www.github.com/logentries - Register your machine - Configure your logs - Monitor Logs - Export Logs What it does: - Push Logs – Communicates with Logentries API - Filter Logs – Uses SSL - Configure your logs (clusters /apps) – Data compressed on the wire - Navigate and Manage your account - Windows Event Logs Other Stuff: – Run as daemon – Run as windows service
- 19. Django • Web Site • Notifications • User Authentication • Landing Pages/Sign up • Billing (Recurly Python Interface) • Application UI
- 20. App Engine Logging Logging on App Engine: • Nice interface • Limited Buffers • Can download logs periodically • No Trend Analysis • No Business Intelligence • App Engine Roadmap - Improvements
- 21. App Engine Logging Logging to Logentries: • In-Process Logging • Push-Queue Logging • Pull-Queue Logging.
- 22. Use Cases & Platforms • Prevent System Crash • Windows/Linux/Mac • Reduce Log Analysis Time • Syslog Forwarding • Business Analytics • Developer Libs • Ruby, Java, C#, Python, • Upload Saved Logs Node.js • Engineyard • Instant Access to All Logs • Heroku • App Engine • Mobile Device Logs • Cloudbees • Secure Log Storage • Mobile Devices • Android
- 23. Logentries
- 24. Logentries
- 25. Logentries
- 26. Logentries
- 27. Logentries
- 28. Logentries
- 29. Questions Get a free account: www.logentries.com