Term Agregation in Kibana: Fielddata is disabled on text fields by default

[AndresL (Andres)]

HI,
In a ESS cluster, Im sending log with Filebeat, and with kibana im trying to create a visualization, Pie Chart by host.hostname.

Then I create a Term agregation by hostname and when i execute the visualization i get the rror in Kibana "1 of 15 shards failed

The data you are seeing might be incomplete or wrong."

This error is related to "Fielddata is disabled". The visualization is correct.
If i check the documentation, enable fielddata can be a negative thing.

What im doing wrong? how to get rid of this message?

Here the error:

{
"took": 258,
"timed_out": false,
"_shards": {
"total": 15,
"successful": 14,
"skipped": 0,
"failed": 1,
"failures": [
{
"shard": 0,
"index": "filebeat-integration-7.3.0-2019.11.04",
"node": "6meXWTYXQiihZaZsg3DBWA",
"reason": {
"type": "illegal_argument_exception",
"reason": "Fielddata is disabled on text fields by default. Set fielddata=true on [host.hostname] in order to load fielddata in memory by uninverting the inverted index. Note that this can however use significant memory. Alternatively use a keyword field instead."
}
}
]
},
"hits": {
"total": 2354,
"max_score": null,
"hits":
},
"aggregations": {
"2": {
"doc_count_error_upper_bound": 0,
"sum_other_doc_count": 0,
"buckets": [
{
"key": "DISIISC08",
"doc_count": 7
},
{
"key": "DISIISC07",
"doc_count": 2347
}
]
}
},
"status": 200
}

[Nathan_Reese]

This error occurs when there is a type mismatch across shards.

In your example, 14 of your 15 shards maps the field host.hostname as keyword. Keywords can be aggregated.

1 of your 15 shards maps the field host.hostname as text. Text fields are used for full text search and can not be aggregated out of the box.

How many Elasticsearch indices does your Kibana Index pattern span? What is the mapping for host.hostname for each Elasticsearch index?

Follow the instructions below if you want to see the problem with minimal noise so you better understand what is going on and figure out how to fix it in your environment.

1. In console create the index test-1 that maps firstname to keyword

PUT test-1
{}

PUT test-1/_mapping
{
  "properties": {
    "firstname": {
      "type": "keyword"
    }
  }
}

PUT test-1/_doc/1
{
    "firstname" : "fred"
}

2. Create index-pattern in Kibana with the title test*

3. In console create the index test-2 that maps firstname to text

PUT test-2
{}

PUT test-2/_mapping
{
  "properties": {
    "firstname": {
      "type": "text"
    }
  }
}

PUT test-2/_doc/1
{
    "firstname" : "wilma"
}

Go to the index-pattern management page for test* and refresh the fields. There is now a mapping conflict between keyword and text for the field "firstname".

Where this becomes a problem is that you can create visualizations on the index-pattern that will produce shard failures because the field is text on some shards and can not be aggregated.

[AndresL (Andres)]

Thanks for your quetsion Nathan, i really understand what happened with your answer
at the begining i was using the default template pattern, but i change the template pattern to something like "filebeat-development-xxxx" or "filebeat-test-xxx", where development and test are environments.

When i was testing a new template patter, i set this configuration to just one shard

setup.template.settings:
index.number_of_shards: 1
setup.template.name: "filebeat"
setup.template.pattern: "filebeat-#{Filebeat_environment}-"