# Auditbeat shards failing **URL:** \ **Category:** Beats\ **Tags:** auditbeat\ **Created:** [August 8, 2019, 12:09pm UTC](https://discuss.elastic.co/t/auditbeat-shards-failing/194428 "2019-08-08T12:09:03Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![nilubkal](https://avatars.discourse-cdn.com/v4/letter/n/d9b06d/32.png) [@nilubkal](https://discuss.elastic.co/u/nilubkal)\ **Post date:** [August 8, 2019, 12:09pm UTC](https://discuss.elastic.co/t/auditbeat-shards-failing/194428/1 "2019-08-08T12:09:03Z")
Hello, since i deployed an ELK 7.2 stack ( 4 days ago ) the auditbeat shards are failing with 3 out of 4 shards when i try to use the defaulf [[Auditbeat Auditd] Overview ECS] dashboard ( or any of the other auditbeat default dashboards).Which leaves me with accessible events only older than 3 days Everything is green in the cluster. Incoming events are indexed correctly but when i execute a query the shards fails, from a data node: ```auto { "type": "server", "timestamp": "2019-08-08T09:53:03,254+0000", "level": "DEBUG", "component": "o.e.a.s.TransportSearchAction", "cluster.name": "elkelasticsearch_name", "node.name": "elkelasticsearch-data-1", "cluster.uuid": "uzybupYsRiuyxlorl0gVfQ", "node.id": "wkLzxc7pSeuhpJqFwAhnsQ", "message": "[auditbeat-2019.08.07][0], node[OM0otphDQ1yK3zRlqJhhKA], [R], s[STARTED], a[id=aAg3q3uXSjSS63M2r3VASQ]: Failed to execute [SearchRequest{searchType=QUERY_THEN_FETCH, indices=[auditbeat-*], indice sOptions=IndicesOptions[ignore_unavailable=false, allow_no_indices=true, expand_wildcards_open=true, expand_wildcards_closed=false, allow_aliases_to_multiple_indices=true, forbid_closed_indices=true, ignore_al iases=false, ignore_throttled=true], types=[], routing='null', preference='null', requestCache=null, scroll=null, maxConcurrentShardRequests=0, batchedReduceSize=512, preFilterShardSize=128, allowPartialSearch Results=true, localClusterAlias=null, getOrCreateAbsoluteStartMillis=-1, ccsMinimizeRoundtrips=true, source={\"size\":0,\"timeout\":\"30000ms\",\"query\":{\"bool\":{\"must\":[{\"range\":{\"@timestamp\":{\"from \":\"2019-08-08T09:52:02.844Z\",\"to\":\"2019-08-08T09:53:02.844Z\",\"include_lower\":true,\"include_upper\":true,\"format\":\"strict_date_optional_time\",\"boost\":1.0}}},{\"query_string\":{\"query\":\"event. module:auditd\",\"fields\":[],\"type\":\"best_fields\",\"default_operator\":\"or\",\"max_determinized_states\":10000,\"enable_position_increments\":true,\"fuzziness\":\"AUTO\",\"fuzzy_prefix_length\":0,\"fuzzy _max_expansions\":50,\"phrase_slop\":0,\"analyze_wildcard\":true,\"escape\":false,\"auto_generate_synonyms_phrase_query\":true,\"fuzzy_transpositions\":true,\"boost\":1.0}}],\"filter\":[{\"match_all\":{\"boost \":1.0}}],\"adjust_pure_negative\":true,\"boost\":1.0}},\"aggregations\":{\"61ca57f1-469d-11e7-af02-69e470af7417\":{\"meta\":{\"timeField\":\"@timestamp\",\"intervalString\":\"1s\",\"bucketSize\":1,\"seriesId\ ":\"61ca57f1-469d-11e7-af02-69e470af7417\"},\"terms\":{\"field\":\"event.action\",\"size\":10,\"min_doc_count\":1,\"shard_min_doc_count\":0,\"show_term_doc_count_error\":false,\"order\":[{\"_count\":\"desc\"}, {\"_key\":\"asc\"}]},\"aggregations\":{\"timeseries\":{\"date_histogram\":{\"field\":\"@timestamp\",\"time_zone\":\"Europe\",\"interval\":\"1s\",\"offset\":0,\"order\":{\"_key\":\"asc\"},\"keyed\":false,\"min_ doc_count\":0,\"extended_bounds\":{\"min\":1565257922844,\"max\":1565257982844}},\"aggregations\":{\"6b9fb2d0-c1bc-11e7-938f-ab0645b6c431\":{\"bucket_script\":{\"buckets_path\":{\"count\":\"_count\"},\"script\ ":{\"source\":\"count * 1\",\"lang\":\"expression\"},\"gap_policy\":\"skip\"}}}}}}}}}] lastShard [true]", "stacktrace": [ "org.elasticsearch.transport.RemoteTransportException: [elkelasticsearch-data-0][10.224.3.247:9300][indices:data/read/search[phase/query]]", "Caused by: java.lang.IllegalArgumentException: Fielddata is disabled on text fields by default. Set fielddata=true on [event.action] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead.", "at org.elasticsearch.index.mapper.TextFieldMapper$TextFieldType.fielddataBuilder(TextFieldMapper.java:711) ~[elasticsearch-7.2.0.jar:7.2.0]", "at org.elasticsearch.index.fielddata.IndexFieldDataService.getForField(IndexFieldDataService.java:116) ~[elasticsearch-7.2.0.jar:7.2.0]", "at org.elasticsearch.index.query.QueryShardContext.getForField(QueryShardContext.java:179) ~[elasticsearch-7.2.0.jar:7.2.0]", "at org.elasticsearch.search.aggregations.support.ValuesSourceConfig.resolve(ValuesSourceConfig.java:95) ~[elasticsearch-7.2.0.jar:7.2.0]", "at org.elasticsearch.search.aggregations.support.ValuesSourceAggregationBuilder.resolveConfig(ValuesSourceAggregationBuilder.java:321) ~[elasticsearch-7.2.0.jar:7.2.0]", "at org.elasticsearch.search.aggregations.support.ValuesSourceAggregationBuilder.doBuild(ValuesSourceAggregationBuilder.java:314) ~[elasticsearch-7.2.0.jar:7.2.0]", "at org.elasticsearch.search.aggregations.support.ValuesSourceAggregationBuilder.doBuild(ValuesSourceAggregationBuilder.java:39) ~[elasticsearch-7.2.0.jar:7.2.0]", "at org.elasticsearch.search.aggregations.AbstractAggregationBuilder.build(AbstractAggregationBuilder.java:139) ~[elasticsearch-7.2.0.jar:7.2.0]", "at org.elasticsearch.search.aggregations.AggregatorFactories$Builder.build(AggregatorFactories.java:332) ~[elasticsearch-7.2.0.jar:7.2.0]", "at org.elasticsearch.search.SearchService.parseSource(SearchService.java:789) ~[elasticsearch-7.2.0.jar:7.2.0]", "at org.elasticsearch.search.SearchService.createContext(SearchService.java:591) ~[elasticsearch-7.2.0.jar:7.2.0]", "at org.elasticsearch.search.SearchService.createAndPutContext(SearchService.java:550) ~[elasticsearch-7.2.0.jar:7.2.0]", "at org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:353) ~[elasticsearch-7.2.0.jar:7.2.0]", "at org.elasticsearch.search.SearchService.lambda$executeQueryPhase$1(SearchService.java:340) ~[elasticsearch-7.2.0.jar:7.2.0]", "at org.elasticsearch.action.ActionListener.lambda$map$2(ActionListener.java:145) ~[elasticsearch-7.2.0.jar:7.2.0]", "at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:62) ~[elasticsearch-7.2.0.jar:7.2.0]", "at org.elasticsearch.search.SearchService$2.doRun(SearchService.java:1052) ~[elasticsearch-7.2.0.jar:7.2.0]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.2.0.jar:7.2.0]", "at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:44) ~[elasticsearch-7.2.0.jar:7.2.0]", "at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:758) ~[elasticsearch-7.2.0.jar:7.2.0]", "at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:37) ~[elasticsearch-7.2.0.jar:7.2.0]", "at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) ~[?:?]", "at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) ~[?:?]", "at java.lang.Thread.run(Thread.java:835) [?:?]" ] } ``` ---
**Author:** ![cwurm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/cwurm/32/34882_2.png) [@cwurm](https://discuss.elastic.co/u/cwurm)\ **Post date:** [August 8, 2019, 1:18pm UTC](https://discuss.elastic.co/t/auditbeat-shards-failing/194428/2 "2019-08-08T13:18:51Z")
It looks like the index `auditbeat-2019.08.07` has the field `event.action` as a `text` field rather than a `keyword` field. Is that an old or new index? If it is a new index, and has been set up using the `./auditbeat setup` command it should be a `keyword` field - so I suspect something went wrong there. If you want to see data from it in the dashboard you can either enable fielddata ([docs](https://www.elastic.co/guide/en/elasticsearch/reference/7.2/fielddata.html#enable-fielddata-text-fields)), or reindex the data with the correct data type for that field. ---
**Author:** ![nilubkal](https://avatars.discourse-cdn.com/v4/letter/n/d9b06d/32.png) [@nilubkal](https://discuss.elastic.co/u/nilubkal)\ **Post date:** [August 9, 2019, 11:24am UTC](https://discuss.elastic.co/t/auditbeat-shards-failing/194428/3 "2019-08-09T11:24:53Z")
Thanks for the help , yep that was the problem, actually i was using a wrong template name and when the logs were rotated the mapping was wrong. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 30, 2019, 11:24am UTC](https://discuss.elastic.co/t/auditbeat-shards-failing/194428/4 "2019-08-30T11:24:57Z")
This topic was automatically closed 21 days after the last reply. New replies are no longer allowed.