Unable to create multiple index in elastic from logstash

[sauravsuman689 (Saurav Suman)]

Hi,

I am using filebeat->logstash->elastic->kibana

I am unable to create multiple index in elastic . Please advise. One index is getting created but not both.

Filebeat.yml is below

---

filebeat.prospectors:

• type: log
  paths:
  • /var/lib/mesos/slave/slaves//frameworks//executors//runs/latest/stdout
  • /var/lib/mesos/slave/slaves//frameworks//executors//runs/latest/stderr
  • /var/log/mesos/*.log
    fields:
    log_type: docker-logs
• type: log
  paths:
  • /nfs/nfsbackup/dcos-stage2/bhw-nginx/logs/access*
  • /nfs/nfsbackup/dcos-stage2/bhw-nginx/logs/error*
    fields:
    log_type: nginx-logs
    fields_under_root: true
    exclude_files: ["stdout.logrotate.state", "stdout.logrotate.conf", "stderr.logrotate.state", "stderr.logrotate.conf"]
    tail_files: true
    output.logstash:
    hosts:
  • "192.168.2.136:5044"
    password: changeme
    username: logstash_system

logstash.conf file under pipeline is below -

input {

     beats {
    port => "5044"
  }

}

filter {
if [log_type] == "nginx-logs" {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}"}
}
}
else if [log_type] == "docker-logs" {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}"}
}
}
}
output {
if [log_type] == "nginx-logs" {
elasticsearch {
hosts => ["192.168.2.191:9201","192.168.2.138:9201","192.168.2.195:9201"]
user => "elastic"
password => "changeme"
index => "web-%{+YYYY.MM.dd}"
}
}
else if [log_type] == "docker-logs" {
elasticsearch {
hosts => ["192.168.2.191:9201","192.168.2.138:9201","192.168.2.195:9201"]
user => "elastic"
password => "changeme"
index => "app-%{+YYYY.MM.dd}"
}
}
}

I can only see web index got created but not the app index. I tried all the ways but no gain.

[Badger]

It looks like you have defined the logstash output under the nginx prospector. You should reduce the level of indentation of it so that output.logstash is at the same level as filebeat.prospectors

[sauravsuman689 (Saurav Suman)]

Hi Badger,

logstash out is under the filebeat.prospectors only. It's just the copy past on the blog which shows it like that. I also validated the filebeat.yml in yml validator online.

[Badger]

The I would enabled debug logging in filebeat.yml. It will tell you which files it is checking, which ones it actually starts prospectors for, and every time it notices a change in a file.

[sauravsuman689 (Saurav Suman)]

Hi Badger,

I enabled debug mode in filebeat.yml file and I see it is pushing both the log_types which I have mentioned in the yml file from it's location.

[Badger]

You have fields_under_root for one but not the other. Either add that or change the test to be

else if [fields][log_type] == "docker-logs" {

[sauravsuman689 (Saurav Suman)]

Hi Badger,

Thanks a lot for the help. It worked and the two indexes got created.

But it's very strange. If you see my filebeat.yml screenshot , I have mentioned fields for both the log type , but it's working for only one of them. Any idea about this. Is there any other way to mention the type under fields section.

[Badger]

For docker-logs you have

fields:
  log_type: docker-logs

whereas for nginx-logs you have

  fields:
    log_type: nginx-logs
  fields_under_root: true

The former results in

"fields": {
  "log_type": "docker-logs"
}.

whereas the latter results in just

"log_type": "nginx-logs"

[sauravsuman689 (Saurav Suman)]

Thank you so much for debugging and explanation..!!