Filebeat not able to create a ES Index

[srijan.nandi]

I have installed elasticsearch, kibana and logstash on 3 separate machines. I am able to use metricbeat, packetbeat and auditbeat without any issue.

But when I install filebeat, it does not create an index on elasticsearch. I have manually loaded the template and I can see the template in ES. But the index does not get created automatically, the way it did for metricbeat and packetbeat. My setup is filbeat to push to logstash and logstash to ES.

I have tried both automatically and manually to put the template.

curl -XPUT -H 'Content-Type: application/json' http://X.X.X.Y:9200/_template/filebeat-6.4.1 -d@filebeat.template.json

My filebeat configuration:

filebeat.inputs:
- type: log

  # Change to true to enable this input configuration.
  enabled: true

  # Paths that should be crawled and fetched. Glob based paths.
  paths:
    - /var/log/secure
    - /var/log/messages

filebeat.config.modules:
  # Glob pattern for configuration loading
  path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  reload.enabled: false

  # Period on which files under path should be checked for changes
  reload.period: 10s

setup.template.settings:
  index.number_of_shards: 1
  index.number_of_replicas: 1
  setup.template.name: "filebeat"
  setup.template.pattern: "filebeat-*"
  setup.template.overwrite: true

output.logstash:
  # The Logstash hosts
  hosts: ["X.X.X.X:5044"]

The template as seen in ES:

curl -XGET http://X.X.X.Y:9200/_template?pretty
{
  ".monitoring-logstash" : {
    "order" : 0,
    "version" : 6040099,
    "index_patterns" : [
      ".monitoring-logstash-6-*"
    ],
    "settings" : {
      "index" : {
        "format" : "6",
        "codec" : "best_compression",

The rest of the indices are seen

health status index                       uuid                   pri rep docs.count docs.deleted store.size pri.store.size
yellow open   auditbeat-6.4.1-2018.09.24  oxhl6enaRDy7q_gACYFlrw   5   1          9            0     91.4kb         91.4kb
yellow open   packetbeat-6.4.1-2018.09.25 otBpda3OQ_uUX8Jr0y66tA   5   1      72681            0     16.4mb         16.4mb
green  open   .kibana                     uIva3SKOSsOhNhRWhOGHXA   1   0         25            4     93.2kb         93.2kb
yellow open   auditbeat-6.4.1-2018.09.23  jt9p5K4hTDC6E-gOSoR4yA   5   1       1476            0   1000.7kb       1000.7kb
yellow open   packetbeat-6.4.1-2018.09.26 4Jcc3xSzSuethsjWVYsl-A   5   1      61922            0     24.2mb         24.2mb
yellow open   auditbeat-6.4.1-2018.09.25  jYBSCT_MQouYKFFtfV3c2w   5   1         16            0      166kb          166kb
yellow open   metricbeat-6.4.1-2018.09.23 ytPYG9YbRaWxYRUAZyvmEA   5   1     128950            0     40.7mb         40.7mb
yellow open   packetbeat-6.4.1-2018.09.24 bEKLAiOPSYSqkygHWmV1rg   5   1      39143            0      9.6mb          9.6mb
yellow open   metricbeat-6.4.1-2018.09.25 Gp3HbEOiSgi7n_pxCFBJyg   1   1     213274            0     33.1mb         33.1mb
yellow open   metricbeat-6.4.1-2018.09.26 Krbp5E_6Qo6jZtOML4aojg   1   1     202551            0       39mb           39mb
yellow open   metricbeat-6.4.1-2018.09.22 kvAq_QlXT0-J1q_rxiUtqw   5   1      35689            0     12.2mb         12.2mb
yellow open   auditbeat-6.4.1-2018.09.26  umGzK1-0Qv-sMLWTKEfC-A   5   1          4            0     38.1kb         38.1kb
yellow open   metricbeat-6.4.1-2018.09.24 6O3vGJAISUCI89iyHkEUWQ   5   1     181624            0     54.7mb         54.7mb

Thanks and Regards

[srijan.nandi]

I now see that an index is being created but with a wrong name. %{[@metadata][beat]}-%{[@metadata][version]}-2018.09.26

yellow open %{[@metadata][beat]}-%{[@metadata][version]}-2018.09.26 2w9SBR2RTq-CcZ_l5UdKVA 5 1 1 0 37.5kb 37.5kb

My logstash output is:

output {
  elasticsearch {
    hosts => ["http://X.X.X.Y:9200"]
    sniffing => true
    manage_template => false
    #index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}" 
    document_type => "%{[@metadata][type]}" 
  }
  stdout { codec => rubydebug }
}