Configuration if [path] =~ "access" not working

Elastic Stack Logstash
1

Hi Team,
I am trying to filter apache logs messages, but its always goes to random_logs than the apache_access or apache_error. When i tried to run as adhoc its parsing properly.

# Logstash configuration
# Beats -> Logstash -> Elasticsearch pipeline.

input {
  beats {
    port => 5044
  }
}
#-------------------------------------------------------------------------------
# Apache log filter
#-------------------------------------------------------------------------------
filter {
  if [path] =~ "access" {
    mutate { replace => { type => "apache_access" } }
    grok {
      match => { "message" => "%{COMBINEDAPACHELOG}" }
    }
    date {
      match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
    }
  } else if [path] =~ "error" {
    mutate { replace => { type => "apache_error" } }
  } else {
    mutate { replace => { type => "random_logs" } }
  }
}
#-------------------------------------------------------------------------------
# Sys log filter
#-------------------------------------------------------------------------------
filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
}
#-------------------------------------------------------------------------------
output {
  if [@metadata][pipeline] {
    elasticsearch {
      hosts => ["http://xxx.xxx.xxx.xx:9200","http://xxx.xxx.xxx.xx:9200","http://xxx.xxx.xxx.xx:9200"]
      manage_template => false
      index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
      pipeline => "%{[@metadata][pipeline]}"
    }
  } else {
    elasticsearch {
      hosts => ["http://xxx.xxx.xxx.xx:9200","http://xxx.xxx.xxx.xx:9200","http://xxx.xxx.xxx.xx:9200"]
      manage_template => false
      index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
    }
  }
}
#-------------------------------------------------------------------------------

My log file path is having log path like below.
/apps/nginx/logs/access.log
/apps/*/access_ssl.log

2

If you are using filebeat I would expect the filename to be in [log][file][path] rather than [path]

3

sorry for the late response, its working perfectly now. But the only issue i have is nginx access.logs also moved as apache_access type. Is there any way to separate that ? nginx path will be /apps/logs/nginx/access.logs

4

Something like

if [path] =~ "nginx/access" {
    mutate { replace => { type => "nginx_access" } }
 } else if [path] =~ "access" {
    mutate { replace => { type => "apache_access" } }
 [...]
5

I gave like below and its putting the nginx logs properly under nginx_access

filter {
  if [log][file][path] =~ "nginx/logs/access" {
    mutate { replace => { type => "nginx_access" } }
  } else if [log][file][path] =~ "access" {
    mutate { replace => { type => "apache_access" } }
    grok {
      match => { "message" => "%{COMBINEDAPACHELOG}" }
    }
    date {
      match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
    }
  } else if [log][file][path] =~ "error" {
    mutate { replace => { type => "apache_error" } }
  } else {
    mutate { replace => { type => "random_logs" } }
  }
}

Is it * will be acceptable in the path check ?

else if [log][file][path] =~ "/apps/*/logs/*/access"

6

It's doing a regexp match, so * means zero or more of the preceding character or class. So "/apps/*/logs/*/access" would match "/apps///logs/access".

You want "/apps/.*/logs/.*/access"

7

Thanks we can close the topic now.

I am using below config which works perfectly.

filter {
  if [log][file][path] =~ "nginx/logs/access" {
    mutate { replace => { type => "nginx_access" } }
  } else if [log][file][path] =~ "nginx/logs/error" {
    mutate { replace => { type => "nginx_error" } }
  } else if [log][file][path] =~ "apps/.*/logs/.*/access" {
    mutate { replace => { type => "apache_access" } }
    grok {
      match => { "message" => "%{COMBINEDAPACHELOG}" }
    }
    date {
      match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
    }
  } else if [log][file][path] =~ "apps/.*/logs/.*/error" {
    mutate { replace => { type => "apache_error" } }
  } else {
    mutate { replace => { type => "random_logs" } }
  }
}
8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.