Beats (6.8.2) via Logstash using collectd index when more than one host specified

[TomD (Tom)]

Hi

If I modify the default config (/etc/filebeat/filebeat.yml) to disable output.elasticsearch and add output.logstash like below I get duplicate updates. One to filebeat-6.8.2-2019.08.08 and another to collectd-2019.08.08

filebeat.inputs:
- type: log
  enabled: true
  paths:
    - /var/log/messages
filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml
  reload.enabled: false
setup.template.settings:
  index.number_of_shards: 3
setup.kibana:
output.logstash:
  hosts: ['logstash01', 'logstash02']
processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~

I use default Logstash configs on both logstash hosts with this /etc/logstash/conf.d/beats.conf as per the documentation

input {
  beats {
    port => 5044
  }
}
output {
  elasticsearch {
    hosts => ["elastic01", "elastic02", "elastic03"]
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
  }
}

Using one logstash host results in the expected behaviour of updates only to the filebeat-* index. I have tried loadbalance: true but it makes no difference

Is this a known issue or am I doing something wrong?

Thanks

[kvch (Noémi Ványi)]

Where did you configure the index collectd-2019.08.08?

[TomD (Tom)]

Hi
I didn't. I don't have collectd publishing anywhere. I delete the index and filebeat/logstash create it again
Thanks
Tom