I want to add a field with filename

Elastic Stack Logstash
1

Hi you all, i'm starting a project with elk stack and we have few domains working on the same machine, which sends his apache logs to elk. I want to extract the filename to filter by this field on kibana.
I'm trying something like the following code:

filter {
  if [type] == "apache-access" {
    grok {
      match => { "message" => "%{COMBINEDAPACHELOG}" }
      match => { "message" => "%{GREEDYDATA}/%{GREEDYDATA:filename}\.es_access_log"}
    }
    mutate {
      remove_field => [ "message" ]
    }
    }
  }

When I look at Kibana I find out that I have a field called source which contains whole path to the file but It's impossible to me to extract filename from here

2

Grok filters stop evaluating expressions when they get a match, so if the COMBINEDAPACHELOG expression matches the other one won't be tried at all. Secondly, if the file path is stored in the source field you obviously need to tell the grok filter to match that field and not message.

3

So, I have to apply another grok filter for this document-type, is this correct?

4

Yes.

5

Thanks for your help, I´m gonna try it.

6

I have been doing tests during this week with this configuration and still does not show the filename, I dont know why is it incorrect.
First config file:

filter {
  if [type] == "apache-access" {
    grok {
      match => { "message" => "%{COMBINEDAPACHELOG}" }
    }
    mutate {
      remove_field => [ "message" ]
    }
    geoip {
        source => "clientip"
    }
  }
}

Second config file:

filter {
    if [%{host}] == "***" {
        grok {
            match => { "source" => "%{GREEDYDATA}/%{GREEDYDATA:filename}\.es_access_log"}
        }
     }
}
7 
if [%{host}] == "***" {

[host], not [%{host}]. I assume "***" is your way of censoring the hostname.

8

Thanks! Now is working properly.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.