topbeat dashboard seems not to like hostnames with "-" character or in filesystem mount points, there's a warning about "-" not being supported because those fields are analyzed fields. Is there a way to change the topbeat index mapping to fix this?
Did you use the topbeat template? https://github.com/elastic/topbeat/blob/master/etc/topbeat.template.json
I did load the topbeat template into ES, wondering if might be because I'm using logstash-YYYY-MM-DD index ... does topbeat need to insert directly into ES ? Or can it go through logstash? I ran into other issues (geo ip) using the default "%{[@metadata][beat]}-%{+YYYY.MM.dd}" index, had to use the logstash- index to get geo working.
current logstash output setup :
output {
elasticsearch {
hosts => "localhost:9200"
sniffing => false
manage_template => true
index => "logstash-%{+YYYY.MM.dd}"
document_type => "%{[@metadata][type]}"
}
}Topbeat data can go through Logstash. The question is to which index you applied the template. Topbeat-* or Logstash-*?
Tried loading to both, I think the issue is my LS is set to send all output to the logstash- index, while the dashboard and topbeat .json index templates are configured to use topbeat-* ,
I configured this setup on a another system using LS -> ES with topbeat sending into LS locally and it all works fine. (using default index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}" LS output to ES)
Seems the issue is where I configured LS to send all output to logstash-* index, that breaks topbeat (since the index templates for beats dashboard and topbeat don't specify logstash-* as the index), but I need logstash-*for geo with filebeat. I imagine there's an easy fix somewhere, just have some confusion about which index to use. I know geo doesn't have the correct type mappings (float or numbers instead of geo point) when using the default index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}" setup in LS output
maybe answering my own question here ... wondering if a simple if condition in my LS config could direct all topbeat data into the topbeat-YYYY.MM.DD index, then all log based data (for geo, etc.) would be directed into the logstash-YYYY.MM.DD index.
I think we are on the right way. A few points here:
- I highly recommend to use different indices for each beat. You can easily write a filter in LS to send each event to the correct index for example based on the type which is sent by each beat.
- As you correctly mentioned, there is no geo in the default template for filebeat, as there is no "geo" value before going to LS. The easiest solution here would be to add the field you use for geo to the template and then apply it: https://github.com/elastic/beats/blob/master/filebeat/etc/filebeat.template.json
Let me know in case it doesn't work with the "routing" inside LS filters.
Thanks @ruflin , this template syntax for geo look correct? for filebeat to properly index geo values?
{
"mappings": {
"_default_": {
"_all": {
"enabled": true,
"norms": {
"enabled": false
}
},
"dynamic_templates": [
{
"template1": {
"mapping": {
"doc_values": true,
"ignore_above": 1024,
"index": "not_analyzed",
"type": "{dynamic_type}"
},
"match": "*"
}
}
],
"properties": {
"@timestamp": {
"type": "date"
},
"message": {
"type": "string",
"index": "analyzed"
},
"offset": {
"type": "long",
"doc_values": "true"
},
"geoip" : {
"dynamic" : "true",
"properties" : {
"ip" : {
"type" : "ip"
},
"latitude" : {
"type" : "float"
},
"location" : {
"type" : "geo_point"
},
"longitude" : {
"type" : "float"
}
}
}
}
}
},
"settings": {
"index.refresh_interval": "5s"
},
"template": "filebeat-*"
}
There is an issue in loading the index patterns in Kibana (using the load.sh script) that overwrites the index pattern in the default search, breaking the logstash-* related dashboards and visualizations.
We seem to be getting this problem as well. The - in the hostname and/or the beat.name makes the dashboard show multiple entries for a beat.name.
1 Like
I've noticed the same problem with the beats-dashboards-1.1.0 release.
It seems like a kibana issue, as in theory the ES data is complete, and the hyphen "-" is just another text character.
Has anyone seen a way to fix this in kibana?
Perhaps its a bug?
I am also having the same issue with beats-dashboards-1.1.1
hostnames containing a hyphen are split as seperate machines on the dashboard.
Any solution for this?
have you applied the index template to elasticsearch before starting topbeat?
Yes, I have.
Did you manage to fix it @Maarten_Van_Damme ?
I'd advise following all the steps I put out in the thread I linked.
Note that you will need to delete ALL filebeat-* indices in that method.
Hi Kareem,
Will test your link today and post the results. thx
You can also look directly at the mapping used by Elasticsearch for an index. With this you can then check to see if hostname is correctly set as not_analyzed.
$ curl http://localhost:9200/topbeat-2016.03.01/_mapping?pretty
"mappings": {
...
"beat": {
"properties": {
"hostname": {
"type": "string",
"index": "not_analyzed",
"ignore_above": 1024
},
...
1 Like
I have it working now.
I stopped all clients from sending data to elasticsearch, erased all elasticsearch data, restarted my clients (topbeat and filebeat services) and then the server names showed up correctly.
Thanks for the help guys, much appreciated.
1 Like