Application Logging With The ELK Stack

Application Logging
With The ELK Stack
@bwaine - #DPC15
Monday, 29 June 15

2
Ben Andersen-Waine
Software Engineer
Contractor
Deployed ELK To Prod
Numerous Times
Monday, 29 June 15

System Logs
Monday, 29 June 15

Application Log
Monday, 29 June 15

Debug Information - Errors (connections,
uncaught exceptions, resource exhaustion)
Narrative Information - Methods Calls,
Event Triggers
Business Events - Purchases, Logins,
Registrations, Unsubscribes
7
Application Log
Monday, 29 June 15

ssh webserver@mydomain.net
tail -f /var/log/nginx/my-site.access.log
tail -f /var/log/my.application.log
ssh data@mydomain.net
tail -f /var/log/mysql/mysql.log
ssh q@mydomain.net
tail -f /var/log/rabbitmq/nodename.log
8
Keeping Track Of All This....
Monday, 29 June 15

9
The Elk Stack
Monday, 29 June 15

1) Monolog
2) Everything else....
11
PHP Logging Tools
Monday, 29 June 15

1) Monolog: Loggers And Handlers
2) Monolog:Tags & Formatters
3) Logging business events
12
Basic Logging Examples
Monday, 29 June 15

use MonologLogger;
use MonologHandlerFingersCrossedHandler;
use MonologHandlerStreamHandler;
$logEnv = getenv('LOG_LEVEL');
$level = empty($logLevel) ? $logEnv : Logger::WARNING;
$appLog = new Logger('AppLog');
$strHandler = new StreamHandler('/var/log/app.log', Logger::DEBUG);
$fcHandler = new FingersCrossedHandler($strHandler, $level);
$appLog−>pushHandler($fcHandler);
$appLog−>debug('LOGGING!');
EG1: Loggers And Handlers
13
Monday, 29 June 15

// Set A Log Level
$logEnv = getenv('LOG_LEVEL');
$level = empty($logLevel) ? $logEnv : Logger::WARNING;
// Create A Logger
14
Monday, 29 June 15

$strHandler
= new StreamHandler('/var/log/app.log', Logger::DEBUG);
$fcHandler
= new FingersCrossedHandler($strHandler, $level);
// Create Handlers
$appLog−>pushHandler($fcHandler);
$appLog−>debug('Start Logging!');
$appLog−>emergency('Something Terrible Happened');
// Push The Handler And Start Logging
15
Monday, 29 June 15

EG 2:Tagging Formatting
$strHandler = new StreamHandler('/var/lg.lg', $level);
$formatter = new LogstashFormatter("helloapp", "application");
$strHandler−>setFormatter($formatter);
$appLog−>pushHandler($strHandler));
$id = $_SERVER('X_VARNISH');
$tag = new TagProcessor(['request−id' => $id])
$appLog−>pushProcessor($tag);
$appLog−>debug("LOGGING!");
16
Monday, 29 June 15

// Create A Logger
$strHandler = new StreamHandler('/var/lg.lg', $level);
$formatter = new LogstashFormatter("helloapp", "app");
// Create A Handler & Formatter
// Set Formatter Onto Handler
$strHandler−>setFormatter($formatter);
$appLog−>pushHandler($strHandler));
//Push Handler Onto Logger
17
Monday, 29 June 15

$id = $_SERVER('X_VARNISH');
$tag = new TagProcessor(['request−id' => $id])
$appLog−>pushProcessor($tag);
$appLog−>debug("LOGGING!");
// Capture A Unique Id, Create A Tag Processor, Push
18
Monday, 29 June 15

2009 - RFC 5424 - Syslog Protocol
Code / Severity
0 Emergency: system is unusable
1 Alert: action must be taken immediately
2 Critical: critical conditions
3 Error: error conditions
4 Warning: warning conditions
5 Notice: normal but signiﬁcant condition
6 Informational: informational messages
7 Debug: debug-level messages
https://tools.ietf.org/html/rfc5424
19
Log Levels
Monday, 29 June 15

2013 - PSR03 - PHP Logging Interface Standard
http://www.php-ﬁg.org/psr/psr-3/
20
PSR3
Monday, 29 June 15

EG 3: Event Logging
use MonologLogger;
use SymfonyComponentEventDispatcherEventDispatcher;
$dispatcher = new EventDispatcher();
$dispatcher−>addListener(
"business.registration.post",
function () use ($busLog) {
$busLog−>info("Customer registered");
}
);
$dispatcher−>dispatch("business.registration.post");
Monday, 29 June 15

Logstash Architecture
1. Logstash Shipper ships logs to
logstash
2. Logstash processes them
3. Logstash Inserts Into Elastic
Search
4. Kibana exposes a web interface
to Elastic Search data
Monday, 29 June 15

Logstash Architecture
Monday, 29 June 15

Why not rate the talk now BEFORE
the demo?
24
https://joind.in/talk/view/14235
Monday, 29 June 15

ELK Demo
25
1) Discover Data (search / diagnose)
2)Visualize Data
3) Produce A Dashboard
4) Demonstrate ‘the new hotness’ of Kibana 4
Monday, 29 June 15

26
https://github.com/LoveSoftware/
getting-started-with-the-elk-stack
Monday, 29 June 15

Logstash Conﬁg
31
Monday, 29 June 15

Logstash Collecting
{
"network": {
"servers": [ "logs.logstashdemo.com:5000" ],
"timeout": 15,
"ssl ca":
"/etc/pki/tls/certs/logstash−forwarder.crt"
},
"files": [
{
"paths": [
"/var/log/nginx/helloapp.access.log"
],
"fields": { "type": "nginx−access" }
}
]
}
32
Monday, 29 June 15

Logstash Processing
input {
lumberjack {
port => 5000
ssl_certificate =>
"/etc/pki/tls/certs/logstash−forwarder.crt"
ssl_key =>
"/etc/pki/tls/private/logstash−forwarder.key"
}
}
Input
33
Monday, 29 June 15

Logstash Processing
Filtering
filter {
if [type] == "nginx−access" {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
date {
match => [ "logdate", "dd/MMM/yyyy:HH:mm:ss Z" ]
}
}
}
34
Monday, 29 June 15

Logstash Processing
Output
output {
elasticsearch { host => localhost }
}
35
Monday, 29 June 15

Groking
grok {
match => { "message" => "%{COMBINEDAPACHELOG}" }
}
https://github.com/elasticsearch/logstash/blob/v1.4.2/patterns/grok-patterns
http://grokdebug.herokuapp.com/
55.3.244.1 GET /index.html 15824 0.043
%{IP:client}
%{WORD:method}
%{URIPATHPARAM:request}
%{NUMBER:bytes}
%{NUMBER:duration}
Monday, 29 June 15

37
Hey Ben....
Have you got time for that
gratuitously ﬂashy geo data demo?
Monday, 29 June 15

Logging Ideas
Release Marker
Error rates of various applications over time
Latency in various percentiles of each application tier
HTTP Responses: 400 series responses
HTTP Responses: 500 series responses
Auto git blame production errors
Auth and Syslogs
39
Monday, 29 June 15

Go Forth And Log....
BUT
Remember log rotation
Beware running out of space
Beware ﬁle logging on NFS
40
Monday, 29 June 15

Questions?
41
Monday, 29 June 15

https://joind.in/talk/view/14235
42
Monday, 29 June 15

Application Logging With The ELK Stack

More Related Content

What's hot (20)

Viewers also liked (6)

Similar to Application Logging With The ELK Stack (20)

More from benwaine (9)

Recently uploaded (20)

Application Logging With The ELK Stack