I have created a Data Table Visualization that gives me the "system.uptime.duration.ms" for 4 Windows servers running Metricbeat 6.5.4. Unfortunately, no matter what timeframe I query, I get "20 Shards out of X Failed" in Kiabana.
My Agents are Windows, my Elasticsearch cluster is 3 RHEL Master nodes with 1 RHEL Logstash Server, and 1 RHEL Kibana Web server with a Coordinating Only node installed. Everything is running 6.5.4.
When I installed everything I loaded the default Metricbeat setup configuration from the agent using ".\metricbeat.exe setup -e -E output.logstash.enabled=false -E output.elasticsearch.hosts=['localhost:9200'] -E setup.kibana.host=localhost:5601" because I am a Logstash user. I don't have xpac installed.
Whenever I get the "20 Shards out of X Failed" message, I see in the Coordinating Nodes logs ...
Caused by: java.lang.IllegalArgumentException: Fielddata is disabled on text fields by default. Set fielddata=true on [beat.hostname] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.
at org.elasticsearch.index.mapper.TextFieldMapper$TextFieldType.fielddataBuilder(TextFieldMapper.java:670) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.index.fielddata.IndexFieldDataService.getForField(IndexFieldDataService.java:115) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.index.query.QueryShardContext.getForField(QueryShardContext.java:166) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.search.aggregations.support.ValuesSourceConfig.resolve(ValuesSourceConfig.java:95) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.search.aggregations.support.ValuesSourceAggregationBuilder.resolveConfig(ValuesSourceAggregationBuilder.java:317) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.search.aggregations.support.ValuesSourceAggregationBuilder.doBuild(ValuesSourceAggregationBuilder.java:310) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.search.aggregations.support.ValuesSourceAggregationBuilder.doBuild(ValuesSourceAggregationBuilder.java:37) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.search.aggregations.AbstractAggregationBuilder.build(AbstractAggregationBuilder.java:139) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.search.aggregations.AggregatorFactories$Builder.build(AggregatorFactories.java:336) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.search.SearchService.parseSource(SearchService.java:807) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.search.SearchService.createContext(SearchService.java:616) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:592) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:367) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.search.SearchService.access$100(SearchService.java:121) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:339) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.search.SearchService$2.onResponse(SearchService.java:335) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.search.SearchService$4.doRun(SearchService.java:1082) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:723) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:41) ~[elasticsearch-6.5.4.jar:6.5.4]
at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-6.5.4.jar:6.5.4]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_191]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_191]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_191]
My logstash output configuration is as follows ....
output {
elasticsearch {
hosts => ["localhost1:9200", "localhost2:9200", "localhost3:9200"]
manage_template => false
index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
}
}
Through previous experimentation and learning, I've found that if I don't have the index template installed properly, I do, I get strange mapping issues and ".keyword" fields created for everything. But this time around, I've done all that. What I've noticed this time around is no ".keyword" fields have been created for aggregation use in my Visualizations. Which I believe what is causing my error because By Default field data is not turned on (and I don't want it turned on) and no ".keyword" fields are being created by the Metricbeat template in the index.
- On a side node, Filebeat and Winglog beat act the same way, no ".keyword" fields for aggregation but I don't get the Shard error with them.
- All of my indices don't have unassigned Shards.
- So, HOW do I turn ".keyword"s on so I can use aggregation in my Visualization properly?
Oddly enough, I am getting the Uptime, I just get this error for some reason. - OR, how do make this error go away?
- Here is the gist to my metricbeat index template from "GET /_template/metricbeat-6.5.4" ...
https://gist.github.com/ajeba99/62490a09f33f3ebf3432e2c0893b9b3a (this is what was installed by the metricbeat setup)
TIA