Unable to create multiple index in elastic from logstash

Elastic Stack Logstash
1

Hi,

I am using filebeat->logstash->elastic->kibana

I am unable to create multiple index in elastic . Please advise. One index is getting created but not both.

Filebeat.yml is below

filebeat.prospectors:

  • type: log
    paths:
    • /var/lib/mesos/slave/slaves//frameworks//executors//runs/latest/stdout
    • /var/lib/mesos/slave/slaves//frameworks//executors//runs/latest/stderr
    • /var/log/mesos/*.log
      fields:
      log_type: docker-logs
  • type: log
    paths:
    • /nfs/nfsbackup/dcos-stage2/bhw-nginx/logs/access*
    • /nfs/nfsbackup/dcos-stage2/bhw-nginx/logs/error*
      fields:
      log_type: nginx-logs
      fields_under_root: true
      exclude_files: ["stdout.logrotate.state", "stdout.logrotate.conf", "stderr.logrotate.state", "stderr.logrotate.conf"]
      tail_files: true
      output.logstash:
      hosts:
    • "192.168.2.136:5044"
      password: changeme
      username: logstash_system

logstash.conf file under pipeline is below -

input {

     beats {
    port => "5044"
  }

}

filter {
if [log_type] == "nginx-logs" {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}"}
}
}
else if [log_type] == "docker-logs" {
grok {
match => { "message" => "%{COMBINEDAPACHELOG}"}
}
}
}
output {
if [log_type] == "nginx-logs" {
elasticsearch {
hosts => ["192.168.2.191:9201","192.168.2.138:9201","192.168.2.195:9201"]
user => "elastic"
password => "changeme"
index => "web-%{+YYYY.MM.dd}"
}
}
else if [log_type] == "docker-logs" {
elasticsearch {
hosts => ["192.168.2.191:9201","192.168.2.138:9201","192.168.2.195:9201"]
user => "elastic"
password => "changeme"
index => "app-%{+YYYY.MM.dd}"
}
}
}

I can only see web index got created but not the app index. I tried all the ways but no gain.

2

It looks like you have defined the logstash output under the nginx prospector. You should reduce the level of indentation of it so that output.logstash is at the same level as filebeat.prospectors

3

Hi Badger,

logstash out is under the filebeat.prospectors only. It's just the copy past on the blog which shows it like that. I also validated the filebeat.yml in yml validator online.

image
image.png1313×488 24 KB

4

The I would enabled debug logging in filebeat.yml. It will tell you which files it is checking, which ones it actually starts prospectors for, and every time it notices a change in a file.

5

Hi Badger,

I enabled debug mode in filebeat.yml file and I see it is pushing both the log_types which I have mentioned in the yml file from it's location.

image
image.png1587×820 96 KB

image
image.png1586×731 66.6 KB

6

You have fields_under_root for one but not the other. Either add that or change the test to be

else if [fields][log_type] == "docker-logs" {
7

Hi Badger,

Thanks a lot for the help. It worked and the two indexes got created.

But it's very strange. If you see my filebeat.yml screenshot , I have mentioned fields for both the log type , but it's working for only one of them. Any idea about this. Is there any other way to mention the type under fields section.

8

For docker-logs you have

fields:
  log_type: docker-logs

whereas for nginx-logs you have

  fields:
    log_type: nginx-logs
  fields_under_root: true

The former results in

"fields": {
  "log_type": "docker-logs"
}.

whereas the latter results in just

"log_type": "nginx-logs"
9

Thank you so much for debugging and explanation..!!