Large Scale Log Analytics with Solr (from Lucene Revolution 2015)
13 likes11,873 views
The document discusses large-scale log analytics using Solr, with a focus on integrating Logstash and Rsyslog for log processing. It includes technical configurations, schema optimizations, and performance tuning tips for using these tools effectively. The content emphasizes flexible setups, efficient maintenance, and strategies for data indexing and querying.
![input {
file {
path => "/opt/logs/example.log"
start_position => "beginning"
…
filter {
grok {
match => [ "message", "%{COMBINEDAPACHELOG}" ]
}
}
output {
solr_http {
…
13
01
Grok](https://image.slidesharecdn.com/lucenerevolution2015-2-151016143435-lva1-app6891/85/Large-Scale-Log-Analytics-with-Solr-from-Lucene-Revolution-2015-13-320.jpg)
![30
01
Normalizing Config
input(type="imfile"
File="/opt/logs/example.log"
Tag="apache")
action(type="mmnormalize"
rulebase="/opt/rsyslog/apache_combined.rb"
)
template(name="json_lines" type="list") {
property(name="$!all-json")
constant(value="n")
}
version=2
rule=:%[
{"type": "word", "name": "clientip"},
{"type": "literal", "text": " "},
...
{"type": "char-to", "name": "agent", "extradata": """},
{"type": "literal", "text": """},
{"type": "rest", "name": "blob"}
]%](https://image.slidesharecdn.com/lucenerevolution2015-2-151016143435-lva1-app6891/85/Large-Scale-Log-Analytics-with-Solr-from-Lucene-Revolution-2015-30-320.jpg)
![rule=apache_combined:%[
{"type": "word", "name": "clientip"},
...
{"type": "char-to", "name": "agent", "extradata": """},
{"type": "literal", "text": """},
{"type": "rest", "name": "blob"}
]%
rule=apache_common:%[
{"type": "word", "name": "clientip"},
...
{"type": "number", "name": "bytes"},
{"type": "rest", "name": "blob", "priority": 65535}
]%
...
33
01
Normalizing with Five Rules
input(type="imfile"
File="/opt/logs/example*"
Tag="apache")
action(type="mmnormalize"
rulebase="/opt/rsyslog/multiple_rules.rb"
)
if $!root <> "" then {
set $.final-json = $!root;
} else {
set $.final-json = $!all-json;
}
template(name="json_lines" type="list") {
property(name="$.final-json") constant(value="n")
}](https://image.slidesharecdn.com/lucenerevolution2015-2-151016143435-lva1-app6891/85/Large-Scale-Log-Analytics-with-Solr-from-Lucene-Revolution-2015-33-320.jpg)
Large Scale Log Analytics with Solr (from Lucene Revolution 2015)
- 1. OCTOBER 13-16, 2016 • AUSTIN, TX
- 2. Large Scale Log Analytics with Solr Rafał Kuć and Radu Gheorghe Sematext Group
- 4. 4 02 Agenda Logstash + Solr rsyslog + Solr rsyslog + Redis + Logstash + Solr Solr
- 7. 7 01 Flow in Logstash /var/log/apache.log redis Rafał @kucrafal grok { "user": "Rafał", "twitter": "@kucrafal" } - w $numberOfWorkers https://cdn2.iconfinder.com/data/icons/gconstruct/2118/gconstruct1-14.png plain {json} input codec filter
- 8. 8 01 Flow in Logstash /var/log/apache.log redis Rafał @kucrafal grok { "user": "Rafał", "twitter": "@kucrafal" } - w $numberOfWorkers https://cdn2.iconfinder.com/data/icons/gconstruct/2118/gconstruct1-14.png workers => 2 plain {json} input codec filter output
- 9. 9 01 Simple Config https://github.com/sematext/lucene-revolution-samples/tree/master/2015 input { file { path => "/opt/logs/example.log" start_position => "beginning" } } output { solr_http { solr_url => "http://localhost:8983/solr/gettingstarted" flush_size => 5000 workers => 4 } } bin/plugin install logstash-output-solr_http apache combined logs
- 11. 11 01 Parse JSON input { file { path => "/opt/logs/example.log.parsed" start_position => "beginning" … filter { json { source => "message" } } output { solr_http { … apache combined logs in JSON bin/logstash -f logstash.conf -w 4 # filterWorkers=4
- 13. input { file { path => "/opt/logs/example.log" start_position => "beginning" … filter { grok { match => [ "message", "%{COMBINEDAPACHELOG}" ] } } output { solr_http { … 13 01 Grok
- 16. 16 01 Flow Options (cont.) http://www.hanselman.com/blog/content/binary/Windows-Live-Writer/ef572a4c3e50_13F7B/redis_logo_a83f44f3-708d-4fad-aa6e-6eb0d6f82001.png https://upload.wikimedia.org/wikipedia/commons/thumb/f/f8/Question_mark_alternate.svg/2000px-Question_mark_alternate.svg.png or Kafka or *MQ or... something light here rsyslog rsyslog rsyslog
- 17. 17 01 Flow in rsyslog /var/log/apache.log syslog socket input
- 18. 18 01 Flow in rsyslog /var/log/apache.log syslog socket main queue (RAM+Disk) input queue.type queue.size ...
- 19. 19 01 Flow in rsyslog /var/log/apache.log syslog socket main queue (RAM+Disk) input queue.type queue.size ... queue.workerThreads (filter, parse and send events)
- 20. 20 01 Flow in rsyslog /var/log/apache.log syslog socket main queue (RAM+Disk) input queue.type queue.size ... queue.workerThreads (filter, parse and send events) queue.dequeueBatchSize
- 21. 21 01 Flow in rsyslog /var/log/apache.log syslog socket main queue (RAM+Disk) input queue.type queue.size ... queue.workerThreads (filter, parse and send events) queue.dequeueBatchSize rsyslog_solr.py rsyslog_solr.py rsyslog_solr.py action template {JSON}
- 22. 22 01 Flow in rsyslog /var/log/apache.log syslog socket main queue (RAM+Disk) input queue.type queue.size ... queue.workerThreads (filter, parse and send events) queue.dequeueBatchSize rsyslog_solr.py rsyslog_solr.py rsyslog_solr.py action template {JSON}
- 23. 23 01 Simple Config (1/2) https://github.com/sematext/lucene-revolution-samples/tree/master/2015 module(load="imfile") module(load="omprog") input(type="imfile" File="/opt/logs/example.log" Tag="apache:") main_queue( queue.highWatermark="100000" queue.lowWatermark="50000" queue.maxDiskSpace="5g" queue.fileName="solr_action" queue.spoolDirectory="/opt/rsyslog/queues" queue.saveOnShutdown="on" queue.workerThreads="4" queue.dequeueBatchSize="500" ) apache combined logs
- 24. 24 01 Simple Config (2/2) template(name="json_lines" type="list" option.json="on") { constant(value="{") constant(value=""timestamp":"") property(name="timereported" dateFormat="rfc3339") constant(value="","message":"") property(name="msg") ... constant(value="","syslog-tag":"") property(name="syslogtag") constant(value=""}n") } action( type="omprog" binary="/opt/rsyslog/rsyslog_solr.py" template="json_lines" ) get from https://github.com/rsyslog/rsyslog/tree/master/plugins/external/solr
- 26. 26 01 Base Result 15% rsyslog, 4x1% rsyslog_solr.py
- 27. 27 01 Base Result 15% rsyslog, 4x1% rsyslog_solr.py 125MB rsyslog, 4x15MB rsyslog_solr.py Depends on queue. Here up to 100K events in RAM
- 28. 28 01 JSON Config # same main queue settings and modules input(type="imfile" File="/opt/logs/example.log.parsed" Tag="apache:") module(load="mmnormalize") action(type="mmnormalize" rulebase="/opt/rsyslog/json.rb" ) template(name="json_lines" type="list") { property(name="$!root") constant(value="n") } action(type="omprog" ... apache combined logs already parsed in JSON version=2 rule=:%root:json%
- 30. 30 01 Normalizing Config input(type="imfile" File="/opt/logs/example.log" Tag="apache") action(type="mmnormalize" rulebase="/opt/rsyslog/apache_combined.rb" ) template(name="json_lines" type="list") { property(name="$!all-json") constant(value="n") } version=2 rule=:%[ {"type": "word", "name": "clientip"}, {"type": "literal", "text": " "}, ... {"type": "char-to", "name": "agent", "extradata": """}, {"type": "literal", "text": """}, {"type": "rest", "name": "blob"} ]%
- 32. 32 01 Normalizing “Should Scale”* sys tem log d -ng performance depends mostly on log length and not on the number of rules: http://blog.gerhards.net/2013/01/performance-of-liblognormrsyslog-parse.html
- 33. rule=apache_combined:%[ {"type": "word", "name": "clientip"}, ... {"type": "char-to", "name": "agent", "extradata": """}, {"type": "literal", "text": """}, {"type": "rest", "name": "blob"} ]% rule=apache_common:%[ {"type": "word", "name": "clientip"}, ... {"type": "number", "name": "bytes"}, {"type": "rest", "name": "blob", "priority": 65535} ]% ... 33 01 Normalizing with Five Rules input(type="imfile" File="/opt/logs/example*" Tag="apache") action(type="mmnormalize" rulebase="/opt/rsyslog/multiple_rules.rb" ) if $!root <> "" then { set $.final-json = $!root; } else { set $.final-json = $!all-json; } template(name="json_lines" type="list") { property(name="$.final-json") constant(value="n") }
- 35. 35 01 OK, so this works: rsyslog rsyslog rsyslog
- 37. 37 01 rsyslog.conf module(load="imfile") module(load="omhiredis") input(type="imfile" File="/opt/logs/example.log" Tag="apache:") template(name="json_lines" type="list" option.json="on") {...} main_queue(queue.workerthreads="1" queue.dequeueBatchSize="100" queue.size="10000") action(type="omhiredis" mode="publish" key="rsyslog_logstash" template="json_lines") ./configure --enable-omhiredis small&light queue
- 38. 38 01 logstash.conf input { redis { data_type => "channel" key => "rsyslog_logstash" batch_count => 100 } } output { solr_http { ... } } JSON codec is implied
- 39. 39 01 Combined Result rsyslog 1% Redis 2% Logstash 200% rsyslog 10MB (10K queue) Redis 1000MB (configurable) Logstash 380MB
- 40. 40 01 5-Rule Normalizing Result rsyslog 100% Redis 2% Logstash 200% rsyslog 30MB Redis 1000MB Logstash 450MB
- 41. 41 01 Shipper conclusions rsyslog rsyslog rsyslog rsyslog rsyslog rsyslog easy setup; flexible heavy light; fast less flexible&easy offloads buffers and Logstash processing; flexible and efficient setup and maintenance overhead
- 42. 42 01 Solr Tuning Agenda Schema and config adjustments Time-based collections Tiered cluster (e.g. hot vs cold nodes)
- 43. 43 01 Schema: Two Kinds of Fields message:failed "docValues": true "omitNorms": true, "omitTermFreqAndPositions": true
- 44. 44 01 Schema: Two Kinds of Fields message:failed "docValues": true "omitNorms": true, "omitTermFreqAndPositions": true +20 to 100% capacity* 10% faster indexing* * http://blog.sematext.com/2014/11/17/solr-presentations-lucene-solr-revolution/
- 45. 45 01 Commits "updateHandler.autoSoftCommit.maxTime": 5000 "updateHandler.autoCommit.maxTime": 60000 <ramBufferSizeMB>200</ramBufferSizeMB> 5s feels near- realtime while searching Flush to disk every minute or 200MB
- 46. 46 01 Commits "updateHandler.autoSoftCommit.maxTime": 5000 "updateHandler.autoCommit.maxTime": 60000 <ramBufferSizeMB>200</ramBufferSizeMB> 5s feels near- realtime while searching Flush to disk every minute of 200MB +10% capacity; 10% faster indexing*
- 47. 47 01 Time-Based Collections indexing, merges, most searches doesn’t change => cache friendly can be optimized delete without triggering merges
- 48. 48 01 Time-Based Collections indexing, merges, most searches doesn’t change => cache friendly => can be optimized delete without triggering merges 20-30x capacity; less indexing degradation* * http://www.slideshare.net/sematext/side-by-side-with-elasticsearch-solr-part-2
- 55. 55 01 Tiered Cluster hot1 hot2 cold1 cold2 cold3 cold4 quick recent searches and indexing rare lengthy requests
- 56. 56 01 Tiered Cluster cold1 cold2 cold3 cold4 quick recent searches and indexing rare lengthy requests hot1 hot2 buffer for indexing spikes
- 57. 57 01 Tiered Cluster cold1 cold2 cold3 cold4 quick recent searches and indexing rare lengthy requests hot1 hot2 buffer for indexing spikes less shards per collection and the cluster is still balanced
- 58. 58 01 Tiered Cluster cold1 cold2 cold3 cold4 quick recent searches and indexing rare lengthy requests hot1 hot2 buffer for indexing spikes less shards per collection and the cluster is still balanced CPU++ RAM++ IO++
- 59. 59 01 Wrap-Up