Grokking multiple line formats

Elastic Stack Logstash
1

New to grokking, and trying to modify an AWS ELB log filter to handle AWS ALB logs (one additional field at the beginning, 4 at the end)

if [type] == "{{logstash_elb_access_logs_type}}" {
grok {
match => ["message", "%{TIMESTAMP_ISO8601:log_timestamp} %{NOTSPACE:elb_name} %{IP:elb_client_ip}:%{INT:elb_client_port:int} (?:%{IP:elb_backend_ip}:%{NUMBER:elb_backend_port:int}|-) %{NUMBER:request_processing_time:float} %{NUMBER:backend_processing_time:float} %{NUMBER:response_processing_time:float} (?:%{INT:elb_status_code:int}|-) (?:%{INT:backend_status_code:int}|-) %{INT:elb_received_bytes:int} %{INT:elb_sent_bytes:int} "(?:%{GREEDYDATA:elb_request}|-)" "(?:%{GREEDYDATA:userAgent}|-)" %{NOTSPACE:elb_sslcipher} %{NOTSPACE:elb_sslprotocol}"]
match => ["message", "%{GREEDYDATA:event_name} for ELB: %{NOTSPACE:elb_name} at %{TIMESTAMP_ISO8601:log_timestamp}"]
}

In that statement, I see two "match" lines. Does grok go through each "match" seeking the best fit, so I simply need to add another Match line, or do I need to create a match that can handle either format?

2

Hi,
Here is a link on grok basics: https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html#_grok_basics

http://grokdebug.herokuapp.com and http://grokconstructor.appspot.com/ are useful for helping building patterns to match your logs.

This is what the match param is: https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html#plugins-filters-grok-match

Grok does go through each match. You will want to create, add a match that catches what you require from your logs.

3

Thank you. Been all through the Grok Basics, and had built some grokkers through the GrokConstructor, just could not find any examples or explanations about how multiple Grok lines were handled. You answered that, and I am good to go.

4

Good to hear. Here is note of break on match which, the first successful match by grok will result in the filter being finished: https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html#plugins-filters-grok-break_on_match

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.