Filebeat not creating new index in Elastic search?

Elastic Stack Beats
1

I have a microservice users-ws for which i want to see logs getting pulled by filebeat and send to elasticsearch and logstash for searching and filtering. I have a problem that with
the configuration below its not creating new indexes ..
filebeat.yml -

filebeat.inputs:
  - type: log
     enabled: true
     paths:
       - C:\my_projects\PhotoAppAPIUsers\users-ws.log

   fields:
     log_type: users-ws-log

filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false



setup.template.settings:
index.number_of_shards: 1

output.elasticsearch:
hosts: ["localhost:9200"]

output.logstash:
hosts: ["localhost:5044"]

logstash conf :

    input {
       beats {
         port => 5044
   }
   }

  filter {
  if [log_type] == "users-ws-log" {
    mutate { add_field => { "[@metadata][target_index]" => "users-ws-log-%{+YYYY.MM}" 
     } }
      } else if [log_type] == "albums-ws-log" {
       mutate { add_field => { "[@metadata][target_index]" => "albums-ws-log-% 
       {+YYYY.MM.dd}" } }
        } else {
         mutate { add_field => { "[@metadata][target_index]" => "unknown-%{+YYYY}" } }
    }
}

output {
  elasticsearch {
    hosts => ["localhost:9200"]
    index => "%{[@metadata][target_index]}"
  }
  stdout { codec => rubydebug}
}

Please let me know if i am missing any thing . Everytime i start my microservice though its generating new logs but i don't see new indexes getting created using filebeat.

Below are the indexes everytime created :
green open ilm-history-1-000001 g8nmTY7UQCiH7VDlSyQyQw 1 0 18 0 25.4kb 25.4kb
yellow open filebeat-7.6.2-2020.04.17-000001 WMENcfDbRsGKNLjwMYIIzw 1 1 7432 0 1.5mb 1.5mb

Also its not creating the indexes the way i have configured in logstash.

2

Hi @pracks1982, welcome to the Elastic community forums!

Looking at your filebeat.yml, I see two outputs enabled — elasticsearch and logstash. This is not supported by Beats; only one output can be enabled at any given time. Are you sure your Filebeat is running without any errors? Could you check the Filebeat log after restarting it?

Shaunak

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.