How to make logstash filter multiple grok pattern

Elastic Stack Logstash
1

Hello, im new to the logstash and i want to know how to make logstash filter multiple grok pattern for example from httpd error logs and access logs, im already have those pattern but logstash seem didnt catch the logs, here is the sample logstash configuration i have

input {
   beats {
    port => 5044
 }
}

filter {
  if "httpd_accesslog" in [fields][type] {
     mutate {
       remove_field => ["log","ecs","input","tags","fields","agent" ,"os"]
       update => {
        "event" => "%{[event][original]}"
        "host" => "%{[host][hostname]}"
        "fields" => "%{[fields][type]}"
        }
    }
    grok {
      break_on_match => false
      match => { "message" => "%{IPORHOST:clientip} %{HTTPDUSER:ident} %{USER:auth} \[%{HTTPDATE:timestamp}\] \"(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})\" %{NUMBER:response} (?:%{NUMBER:bytes}|-)" }
    }
  }
   else if "httpd_errorlog" in [fields][type] {
     mutate {
       remove_field => ["log","ecs","input","tags","fields","agent" ,"os"]
       update => {
        "event" => "%{[event][original]}"
        "host" => "%{[host][hostname]}"
        "fields" => "%{[fields][type]}"
        }
    }
    grok {
      break_on_match => false
      match => { "message" => "%{HTTPD20_ERRORLOG}|%{HTTPD24_ERRORLOG}" }
    }
  }
}

output {

    if "httpd_accesslog" in [fields] {
        solr_http {
            solr_url => "http://ip address:8983/solr/access-logs"
        }
    }
    else if "httpd_errorlog" in [fields] {
        solr_http {
            solr_url => "http://ip address:8983/solr/error-logs"
        }
    }
}

and here is my filebeat.yml file

filebeat.inputs:

- type: log
  enabled: true
  paths:
    - /var/log/httpd/access_log
  fields:
    type: httpd_accesslog

- type: log
  enabled: true
  paths:
    - /var/log/httpd/error_log
  fields:
    type: httpd_errorlog
  multiline.type: pattern
  multiline.pattern: '^\['
  multiline.negate: true
  multiline.match: after
  multiline.max.lines: 30

# ------------------------------ Logstash Output -------------------------------
output.logstash:
  # The Logstash hosts
  hosts: ["ipaddress:5044"]
2

In the filter section you have if "httpd_accesslog" in [fields][type] {, which is testing whether that string occurs with that field (a substring match). That will probably work, but I would suggest an equality test, which I think is clearer

if [fields][type] == "httpd_accesslog"

In the output section you have if "httpd_accesslog" in [fields] {, which I do not think will ever work because [fields] is an object, not a string. Use the same test I suggested for the filter section.

1 Like
4

Thank you so much @Badger it's works

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.