Not able to create indexes using multiple logs files in filebeat tags

Elastic Stack Beats
1

Hi Guys,

I'm not able to create custom indexes in elastic search using logstash and filebeat.
Log flow (Filebeat ==> logstash ==> elasticsearch)

I have two custom logs files such as /var/log/app1/app1.log and /var/log/app2/app2.log.
In Filebeat I want to pass these log files along with tags, so each log file has its own tags i.e (app1 and app2).
Using these tags, I want to filter it out using logstash, so it can create two custom indexes in elastic search (app1_logs and app2_logs).
Below is my config file, please suggest to me if any changes are required.

filebeat.yml

filebeat.inputs:

  • type: log
    enabled: true
    paths:

    • var/log/app1/app1.log
      fields:
      tags: ["app1"]

  • type: log
    enabled: true
    paths:

    • /var/log/app2/app2.log
      fields:
      tags: ["app2"]

demo-logstash.conf

input {
beats {
type => "log"
port => 5044
host => "0.0.0.0"
}
}

output {
if [fields][tags] == "app1" {
elasticsearch {
hosts => ["http://localhost:9200"]
index => "app1_logs"
}

}
if [fields][tags] == "app2" {
elasticsearch {
hosts => ["http://localhost:9200"]
index => "app2_logs"
}

}
stdout { codec => rubydebug }
}

2

It's because the tags key is an array. Try using index 0 of the array for the if statement or use a field that isn't an array

3

Hi @legoguy1000 ,

I have tried the below one, but not able to create indexes.
Can you please share an example?

if [fields][tags][0] == "app1" {
elasticsearch {
hosts => ["http://localhost:9200"]
index => "app1_logs"
}
}
if [fields][tags][0] == "app2" {
elasticsearch {
hosts => ["http://localhost:9200"]
index => "app2_logs"
}
}

4

Hi @legoguy1000 ,

The problem has been solved !!!
Instead of using log type in filebeat I have used filestream type and it's start working for me.

Before:
filebeat.yml

  • type: log
    enabled: true
    paths:
    /var/log/app1/app1.log
    fields:
    tags: ["app1"]

After:
filebeat.yml

  • type: filestream
    enabled: true
    paths:
    /var/log/app1/app1.log
    fields:
    tags: ["app1"]

demo-logstash.conf

output {
if [fields][tags][0] == "app1" {
elasticsearch {
hosts => ["http://localhost:9200"]
index => "app1_logs"
}

}

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.