Grok pattern or stanza for parsing elasticsearch slow logs?

Elastic Stack Logstash
1

Any existing grok patterns (or existing grok stanza) for elasticsearch logs? In particular the slow logs? (index_indexing and index_search)?

These are the logs located in a typical install in /var/log/elasticsearch

There is a tutorial here but it is out of date and does not work. (http://blog.sematext.com/2015/01/19/grok-elasticsearch-logs-with-logstash/)

2

We're hoping to get json logging into ES (I'm hoping for 2.0 :p) which will negate this.

But I haven't seen anything to help sorry.

3

We use these patterns:
index_search: "[%{TIMESTAMP_ISO8601:logtime}][%{LOGLEVEL:log_level}\s*][%{DATA:es_slowquery_type}]\s*[%{DATA:es_host}]\s*[%{DATA:es_index}]\s*[%{DATA:es_shard}]\stook[%{DATA:es_duration}],\stook_millis[%{DATA:es_duration_ms:float}],\stypes[%{DATA:es_types}],\sstats[%{DATA:es_stats}],\ssearch_type[%{DATA:es_search_type}],\stotal_shards[%{DATA:es_total_shards:float}],\ssource[%{GREEDYDATA:es_source}],\sextra_source[%{GREEDYDATA:es_extra_source}],"

index_indexing: "[%{TIMESTAMP_ISO8601:log_time}][%{LOGLEVEL:log_level}\s*][%{DATA:es_slowquery_type}]\s*[%{DATA:es_host}]\s*[%{DATA:es_index}]\s*[%{DATA:es_shard}]\stook[%{DATA:es_duration}],\stook_millis[%{DATA:es_duration_ms:float}],\stype[%{DATA:es_type}],\sid[%{DATA:es_id}],\srouting[%{DATA:es_routing}],\ssource[%{GREEDYDATA:es_source}]"

3 Likes
4

A long overdue thanks for the update @warkolm and especially for sharing your pattern @cschotke!