How to customize index name with multipule inputs

Elastic Stack Logstash
1

As suggested by @magnusbaeck I create a new topic about customized index name with multiple inputs.

Case :

I have 2 Apache logs from 2 projects which are stored separately in the directories /tmp/toto/first and /tmp/toto/second. Now, I want to have different index names distinguished by the project name (first and second). In Apache logs, there's no information about where it is stored (path information). I use Filebeat + LogStash + ES + Kibana with the latest version 5.2.1

What I do :

My configuration is as below

Filebeat :

...
  paths:
    - /tmp/toto/*/*.log
...

LogStash :

input {
    beats {
        port => "5043"
    }
}

filter {
    grok {
        match => { "paths" => "/tmp/toto/(?<project>[^/]+)/" }
        match => { "message" => "%{COMBINEDAPACHELOG}" }
    }
    date {
    match => [ "timestamp" , "dd/MMM/yyyy:HH:mm:ss Z" ]
  }

}
output {
    elasticsearch {
        hosts => [ "127.0.0.1:9200" ]
        index => [ "log-%{project}-%{+YYYY.MM.dd}" ]
    }
}

My question :

My index name is shown as log-%{project}-2017.02.22 . My question is in my case, is it possible to have customized index ? Could grok unterstand the path information by key value paths in Filebeat ? How does Filebeat transfer log information to message to let gork understand ? Is there a list of elements like message in grok to match ?

Thanks in advance.

2

Have you checked to see if the file path information is in a subfield of the @metadata field? You can see this field's contents if you add an output like:

output {
  stdout {
    codec => rubydebug { metadata => true }
  }
}

Since you are shipping with beats, this may be the way to find if that's included in the metadata.

For reference, Logstash keeps a @metadata field which does not get attached to outbound data. Different beats ship to this @metadata field in case it is desired for situations such as these. Unfortunately, the filebeat documentation does not reveal what, if any, default information goes into the metadata.

3

Actually, it appears that @metadata doesn't enter into it. You can find the file path in the source field in filebeat.

4

@theuntergeek

Hi aaron, thanks for your reply, it's very useful.

After checking the output, i found the filebeat well did his work to ship to logstash, so i just modify the key word paths to source and it works.

Thanks again for the help.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.