Getting field data error in Kibana dashboards

[tcouto]

I'm getting the following error when trying to view my Kibana dashboards after reaching the storage limit in an elastic cloud instance.

Visualize: Fielddata is disabled on text fields by default. Set fielddata=true on [RequestURI] in order to load fielddata in memory by uninverting the inverted index. Note however that this can use significant memory.

This is occurring for multiple fields and only started occurring after reaching the storage limit, deleting indices, and rebooting the cluster.

As far as I know, doc_values allow you to aggregate a field for visualizations, and they are enabled by default on all fields except for analyzed strings. I can see from the get_mapping requests that none of my fields have doc_values disabled. Why is Kibana throwing this error? Why would I need field_data when I should have doc_values? And how can I fix this?

Thanks,
Tyler

[Stacey_Gammon]

Interesting.

What field is throwing this error? Is this happening when you are creating a visualization with the field, or only when you view an existing visualization? If existing, can you create a new visualization with that field? Is it a text field or a numeric field? (Text fields are analyzed strings so they won't have field_data).

Can you send a screenshot of the following screen in management of the field that is causing the issue? I want to narrow down where the error is coming from.

[tcouto]

Hi, Stacey. Thanks for replying. There a several fields that are throwing this error, and they're all text fields. RequestURI is one. The following is a snippet from the get_mapping call from the Python ElasticSearch library.

          "RequestURI": {
            "fields": {
              "keyword": {
                "ignore_above": 256,
                "type": "keyword"
              }
            },
            "type": "text"
          },

This is happening for new and existing visualizations. I'm adding screenshots for the mapping screen and when I'm trying to add a new visualization.

[Stacey_Gammon]

Hmm, strange. According to the _mapping call, it appears that RequestURI is an analyzed text field and not aggregatable, but you should have a RequestURI.keyword field that is aggregatable.

Does the field RequestURI.keyword show up as available when creating the visualization?

However that's not what kibana is showing in the index field list. I bet this is a discrepancy between the mapping and the field_stats api (kibana gets it's information from the field_stats api).

Can you show the output of GET /_field_stats?fields=RequestURI and GET /_field_stats?fields=RequestURI.keyword? (I'm running these queries in the kibana Dev Tools section)

[tcouto]

I do not see RequestURI.keyword as an option when splitting by term.

Here's the result of GET /_field_stats?fields=RequestURI.keyword

{
  "_shards": {
    "total": 1833,
    "successful": 1817,
    "failed": 0
  },
  "indices": {
    "_all": {
      "fields": {
        "RequestURI.keyword": {
          "type": "string",
          "max_doc": 163149910,
          "doc_count": 42726,
          "density": 0,
          "sum_doc_freq": 42726,
          "sum_total_term_freq": -1,
          "searchable": true,
          "aggregatable": true,
          "min_value": "some_min_request_uri",
          "max_value": "some_maximum_request_uri"
        }
      }
    }
  }
}

Here's the result of GET /_field_stats?fields=RequestURI

{
  "_shards": {
    "total": 1833,
    "successful": 1817,
    "failed": 0
  },
  "indices": {
    "_all": {
      "fields": {
        "RequestURI": {
          "type": "string",
          "max_doc": 288190396,
          "doc_count": 75003,
          "density": 0,
          "sum_doc_freq": 549105,
          "sum_total_term_freq": -1,
          "searchable": true,
          "aggregatable": true,
          "min_value": "0",
          "max_value": "yh_singpore"
        }
      }
    }
  }
}

I hope you don't mind me sanitizing the min and max values for "RequestURI.keyword".

[tcouto]

I can view data in visualizations if I extend the timeframe back enough, so maybe RequestURI.keyword is not being added to new documents? What could cause this?

I also see something that looks strange, and I'm not sure if this is expected. When I use the get_field_mapping call it seems like there are too many levels. I have a mapping under my field.

res = es.indices.get_field_mapping(fields = ['RequestURI'], index = ['certain-applog-2017.02.27'])

{
  "certain-applog-2017.02.27": {
    "mappings": {
      "CertainAppThread": {
        "RequestURI": {
          "mapping": {
            "RequestURI": {
              "fields": {
                "keyword": {
                  "ignore_above": 256,
                  "type": "keyword"
                }
              },
              "type": "text"
            }
          },
          "full_name": "RequestURI"
        }
      }
    }
  }
}

res = es.indices.get_mapping(index = ['certain-applog-2017.02.27'])

"RequestURI": {
  "fields": {
    "keyword": {
      "ignore_above": 256,
      "type": "keyword"
    }
  },
  "type": "text"
},

[Stacey_Gammon]

The get_field_mapping result is okay, that's what mine looks like too (nested like that).

I wasn't able to repro the error you are seeing in Kibana, but I was able to get my index in a weird state like you have. I can do this by creating an index test-index-1 with a mapping like you are seeing:

PUT test-index-1/_mapping/type
{
  "properties": {
    "RequestURI": {
      "fields": {
        "keyword": {
          "ignore_above": 256,
          "type": "keyword"
        }
      },
      "type": "text"
    }
  }
}

Then I create another index with a mapping where RequestURI is aggregatable:

PUT test-index-2/_mapping/type
{
  "properties": {
   "RequestURI": {
    "ignore_above": 256,
    "type": "keyword"
  }
 }
}

Now if I query GET /_field_stats?fields=RequestURI I get aggregatable: true even though it's only true for a subset of indexes.

Trying to create a visualization on this field throws an error (albeit not the error you were seeing, but my test index wasn't time based... maybe that affected it).

So my guess is your mapping changed at some point, for specific indexes. When you query the mapping, what happens if you change the specific index to just certain-applog-*? Do you see more than one result?

When I query GET test-*/_mapping/field/RequestURI the result is:

"test-index-1": {
    "mappings": {
      "type": {
        "RequestURI": {
          "full_name": "RequestURI",
          "mapping": {
            "RequestURI": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            }
          }
        }
      }
    }
  },
  "test-index-2": {
    "mappings": {
      "type": {
        "RequestURI": {
          "full_name": "RequestURI",
          "mapping": {
            "RequestURI": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        }
      }
    }
  }

One for each index.

What is your index pattern in kibana? If you set it explicitly to 'certain-applog-2017.02.27' do you then see the RequestURI.Keyword field?

Where are you getting your data from? Perhaps your configuration files changed at a certain point so that all indexes after time X are using a new mapping configuration?

[tcouto]

Yes, when I query for certain-applog-* a get one for each index. They're not all different, and for the most part the differences only have to do with whether a log file was indexed that day. But this is interesting. When I query each index separately (I only have 5 right now since we deleted so many) the only one that has a _default_ mapping is the one from Feb. 21. The rest are missing the following section.

  "certain-applog-2017.02.21": {
    "mappings": {
      "_default_": {
        "_all": {
          "norms": false,
          "enabled": true
        },
        "properties": {
          "geoip": {
            "dynamic": "true",
            "properties": {
              "location": {
                "type": "geo_point"
              }
            }
          },
          "@timestamp": {
            "type": "date"
          },
          "message": {
            "type": "text"
          },
          "offset": {
            "type": "long"
          }
        },
        "dynamic_templates": [
          {
            "template1": {
              "mapping": {
                "ignore_above": 1024,
                "doc_values": true,
                "type": "{dynamic_type}",
                "index": "not_analyzed"
              },
              "match": "*"
            }
          }
        ]
      },

If I set my index pattern to certain-applog-2017.02.28, I do see RequestURI.keyword

And here's the one from the 21st.

All of our data is from log files shipped from logstash. We don't usually define an index template on the logstash side, but I'll definitely have to dig into how that could have changed.

When I limit my visualization index pattern to certain-applog-2017.02.28, I can see all the .keyword varieties of my fields and aggregate on them.

Thanks so much for your help! I'll definitely have to dig in to our index templates on my own. Have you ever seen index patterns change on the ElasticSearch side? Could they have changed when we ran out of storage space?

[Stacey_Gammon]

Perhaps it was due to an upgrade issue? Were you indexing data prior to v5.0? The default mapping of strings to both keyword and text fields was new in 5.0:

Elasticsearch replaces string type with two new types text and keyword.

[tcouto]

Thanks for that link. I think we upgraded to Elasticsearch 5 a least a month ago, and we were indexing before that, so I'll check that as well.

Ah, here it is. I should've seen this before. Here is the RequestURI mapping in the template from 02/21.

"RequestURI": {
  "ignore_above": 1024,
  "type": "keyword"
},

And here's the mapping from 02/25.

"RequestURI": {
  "fields": {
    "keyword": {
      "ignore_above": 256,
      "type": "keyword"
    }
  },
  "type": "text"
},

The 25th is when our cluster ran out of space, and we had to delete indexes. Maybe during that time, our index templates were also deleted, so they were regenerated in the 5.x default format. Does that make sense?

Anyway, I think if I delete the oldest index, then I'll be able to visualize on the .keyword fields.

I had to refresh my index pattern in the management -> index patterns page. I imagine if I had done this before I removed the indexes with old index templates, it would have worked. But now it's clean! I'll just have to update my visualizations with the .keyword forms of the terms.

Thanks again!