How to customize index name with multipule inputs

As suggested by @magnusbaeck I create a new topic about customized index name with multiple inputs.

Case :

I have 2 Apache logs from 2 projects which are stored separately in the directories /tmp/toto/first and /tmp/toto/second. Now, I want to have different index names distinguished by the project name (first and second). In Apache logs, there's no information about where it is stored (path information). I use Filebeat + LogStash + ES + Kibana with the latest version 5.2.1

What I do :

My configuration is as below

Filebeat :

...
  paths:
    - /tmp/toto/*/*.log
...

LogStash :

input {
    beats {
        port => "5043"
    }
}

filter {
    grok {
        match => { "paths" => "/tmp/toto/(?<project>[^/]+)/" }
        match => { "message" => "%{COMBINEDAPACHELOG}" }
    }
    date {
    match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
  }

}
output {
    elasticsearch {
        hosts => [ "127.0.0.1:9200" ]
        index => [ "log-%{project}-%{+YYYY.MM.dd}" ]
    }
}

My question :

My index name is shown as log-%{project}-2017.02.22 . My question is in my case, is it possible to have customized index ? Could grok unterstand the path information by key value paths in Filebeat ? How does Filebeat transfer log information to message to let gork understand ? Is there a list of elements like message in grok to match ?

Thanks in advance.

Have you checked to see if the file path information is in a subfield of the @metadata field? You can see this field's contents if you add an output like:

output {
  stdout {
    codec => rubydebug { metadata => true }
  }
}

Since you are shipping with beats, this may be the way to find if that's included in the metadata.

For reference, Logstash keeps a @metadata field which does not get attached to outbound data. Different beats ship to this @metadata field in case it is desired for situations such as these. Unfortunately, the filebeat documentation does not reveal what, if any, default information goes into the metadata.

Actually, it appears that @metadata doesn't enter into it. You can find the file path in the source field in filebeat.

@theuntergeek

Hi aaron, thanks for your reply, it's very useful.

After checking the output, i found the filebeat well did his work to ship to logstash, so i just modify the key word paths to source and it works.

Thanks again for the help.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.