I was running filebeat in my cluster for a while now and using elastic to ingest the output without a problem but have a business need to do some filtering and decided to change the output to now use logstash instead. Updated my configs and all seems to be working out fine however I now understand that filebeat, when pointing to an elastic output, will automatically load the filebeat elastic template. Ok, lesson learned.
My problem is that my current index has multiple fields like beat.name and beat.name.keyword, which it seems the latter is being used since using logstash as my filebeat output. A side effect is that my loaded dashboards no longer work because they are aggregating on beat.name and not beat.name.keyword. Mind you, its more than just beat.name and just using it as an example.
Yesterday, I decided to remove and rebuild my filebeat index and manually load my templates on each of my cluster hosts but that didnt seem to work out the way I expected.
filebeat setup --template -E output.logstash.enabled=false -E 'output.elasticsearch.hosts=["localhost:9200"]'
Aside from changing all of my dashboard to now look at x.keyword, is there a better way of doing this?
I'm testing the following at the end of my filebeat.yml config but that doesn't seem to be working either
output.logstash:
compression_level: 1
setup.template.enabled: true
index: filebeat-6*
enabled: true
hosts: [{{ filebeat_settings.hosts }}]
timeout: 90
loadbalance: true
Here's what I think is relevant from my filebeat template settings via GET /_template/filebeat-6*?pretty:
"beat": {
"properties": {
"timezone": {
"type": "keyword",
"ignore_above": 1024
},
"version": {
"type": "keyword",
"ignore_above": 1024
},
"name": {
"type": "keyword",
"ignore_above": 1024
},
"hostname": {
"type": "keyword",
"ignore_above": 1024
}
}
},
Anyone have any easy fixes for this?