Running out of fielddata without enabling it

Elastic Stack Elasticsearch
1

I have read that field data is disabled by default on text fields now. I recently experienced a field data related crash, but we have not even explicitly enabled it in any of our mappings. Are there other ways that elasticsearch could be using field data at any given time??

We are using elasticsearch 5.6

2

How do you know this?

What does _cat/fielddata show?

3 
curl IPADDRESS:9200/_cat/fielddata:
QwUELir0Sn24kggnuEHU8w IPADDRESS IPADDRESS DATA_NODE_NAME _uid 6.8gb
bRuBIe6fS2qtHmLHQnJpkQ IPADDRESS IPADDRESS DATA_NODE_NAME _uid 7.4gb
NAdsL0VfQwezpdZVHU01gQ IPADDRESS  IPADDRESS  DATA_NODE_NAME _uid   7gb
Yg9WcZOhQhWCQ6PG6UHP3g IPADDRESS IPADDRESS DATA_NODE_NAME _uid   7gb
tu8ioPFvQYGU1uJ6OQRrwg IPADDRESS  IPADDRESS  DATA_NODE_NAME _uid   7gb
4

What sort of data is it? What does the mapping look like?

5

Apologize, I didn't see your first question about how we knew it, the cluster didnt actually crash, we started seeing queries fail because we were out of fielddata, and we knew because our application was throwing the error Error: fielddata too large. We cleared the fielddata cache and everything returned to normal. We are already looking at setting smarter values for our field data cache size and breaker values.

I didn't realize that the fielddata gave a breakdown by actual field, however all of these usages point to the "_uid" field, which is not even a field that exists in our mapping.

It is metadata, things like file names and permissions.

this is the mapping:

6 
{
  "assets_v1" : {
    "mappings" : {
      "asset" : {
        "properties" : {
          "bc_asset_id" : {
            "type" : "keyword"
          },
          "checksum" : {
            "type" : "keyword"
          },
          "created_date" : {
            "type" : "long"
          },
          "directory_owner_composite_name" : {
            "type" : "text",
            "fields" : {
              "keyword" : {
                "type" : "keyword",
                "ignore_above" : 10000
              }
            }
          },
          "directory_owner_display_name" : {
            "type" : "text",
            "fields" : {
              "keyword" : {
                "type" : "keyword",
                "ignore_above" : 10000
              },
              "keyword_lowercase" : {
                "type" : "keyword",
                "ignore_above" : 10000,
                "normalizer" : "keyword_lowercase"
              }
            }
          },
          "directory_owner_email" : {
            "type" : "text",
            "fields" : {
              "keyword" : {
                "type" : "keyword",
                "ignore_above" : 10000
              }
            }
          },
          "directory_owner_external_id" : {
            "type" : "text",
            "fields" : {
              "keyword" : {
                "type" : "keyword",
                "ignore_above" : 10000
              }
            }
          },
          "directory_owner_is_deleted" : {
            "type" : "boolean"
          },
          "directory_owner_type" : {
            "type" : "long"
          },
          "download_url" : {
            "type" : "keyword"
          },
          "external_asset_id" : {
            "type" : "keyword"
          },
          "file_size_bytes" : {
            "type" : "long"
          },
          "ingestion_timestamp" : {
            "type" : "long"
          },
          "integration_id" : {
            "type" : "keyword"
          },
          "is_deleted" : {
            "type" : "boolean"
          },
          "is_directory" : {
            "type" : "boolean"
          },
          "max_visibility" : {
            "type" : "long"
          },
          "mime_type" : {
            "type" : "text",
            "fields" : {
              "keyword" : {
                "type" : "keyword",
                "ignore_above" : 10000
              },
              "keyword_lowercase" : {
                "type" : "keyword",
                "ignore_above" : 10000,
                "normalizer" : "keyword_lowercase"
              }
            }
          },
          "org_unit_name" : {
            "type" : "text",
            "fields" : {
              "keyword" : {
                "type" : "keyword",
                "ignore_above" : 10000
              },
              "keyword_lowercase" : {
                "type" : "keyword",
                "ignore_above" : 10000,
                "normalizer" : "keyword_lowercase"
              }
            }
          },
          "permission_entities" : {
            "type" : "nested",
            "properties" : {
              "bc_permission_id" : {
                "type" : "keyword"
              },
              "external_permission_id" : {
                "type" : "keyword"
              },
              "is_downloadable" : {
                "type" : "boolean"
              },
              "is_restricted" : {
                "type" : "boolean"
              },
              "shared_directory_composite_name" : {
                "type" : "text",
                "fields" : {
                  "keyword" : {
                    "type" : "keyword",
                    "ignore_above" : 10000
                  }
                }
              },
              "shared_directory_display_name" : {
                "type" : "text",
                "fields" : {
                  "keyword" : {
                    "type" : "keyword",
                    "ignore_above" : 10000
                  },
                  "keyword_lowercase" : {
                    "type" : "keyword",
                    "ignore_above" : 10000,
                    "normalizer" : "keyword_lowercase"
                  }
                }
              },
              "shared_directory_email" : {
                "type" : "text",
                "fields" : {
                  "keyword" : {
                    "type" : "keyword",
                    "ignore_above" : 10000
                  }
                }
              },
              "shared_directory_owner_type" : {
                "type" : "long"
              },
              "shared_directory_target_id" : {
                "type" : "keyword"
              },
              "type" : {
                "type" : "long"
              },
              "visibility_level" : {
                "type" : "long"
              }
            }
          },
          "provider_id" : {
            "type" : "keyword"
          },
          "raw" : {
            "type" : "text"
          },
          "relative_path" : {
            "type" : "text",
            "fields" : {
              "keyword" : {
                "type" : "keyword",
                "ignore_above" : 10000
              },
              "keyword_lowercase" : {
                "type" : "keyword",
                "ignore_above" : 10000,
                "normalizer" : "keyword_lowercase"
              }
            }
          },
          "tenant_id" : {
            "type" : "keyword"
          },
          "title" : {
            "type" : "text",
            "fields" : {
              "keyword" : {
                "type" : "keyword",
                "ignore_above" : 10000
              },
              "keyword_lowercase" : {
                "type" : "keyword",
                "ignore_above" : 10000,
                "normalizer" : "keyword_lowercase"
              }
            }
          },
          "updated_date" : {
            "type" : "long"
          }
        }
      }
    }
  }
}
7

_uid = https://www.elastic.co/guide/en/elasticsearch/reference/5.6/mapping-uid-field.html

What do the queries look like?

8

We found the issue, thank you @warkolm!! The queries were not the issue, however it turns out we were sorting on _uid. We were able to change our sorts to be on a different field without affecting anything else and our fielddata usage dropped to 0.

1 Like
9

Thanks for sharing the solution!

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.