Find documents that might have unindexed fields

We recently disabled dynamic field mapping on one of our logging index templates after a field explosion basically took down our cluster.

Most documents (log lines) coming into the indices should only have fields that are part of the index template, but we'd like to check and see if there are any documents coming in that are now missing fields because dynamic mapping is off.

Is it possible to do a search for documents that have some unindexed/unmapped fields? Thanks!

Daniel,
I don't think you can use standard ES queries to find these documents.

It will be slow but you can brute force it using scripted fields query. script will have access to _source. So you can get all keys (field names) and emit those that are not in your field list. Note you will get all logs entries in results. In the calling code you will have to filter out docs with not-null value for the scripted fields.

It's not an elegant solution but it will save you from fetching _source for all docs in the client.

Thanks @Vinayak_Sapre, I'll have to do some reading and give that a shot.

It turns out that this was pretty easy after doing some learning about script fields. I was able to write two queries:
One that just returns a list of all fields in _source per document

{
  "query": {
    "bool": {
      "must": [
        {
          "range": {
            "@timestamp": {
              "gte": "now-10m",
              "lte": "now"
            }
          }
        }
      ]
    }
  },
  "script_fields": {
    "field_names": {
      "script": {
        "source": "params._source.keySet()"
      }
    }
  }
}

And a terms aggregation that returns one list of all fields seen in _source

{
  "size": 0,
  "query": {
    "bool": {
      "must": [
        {
          "range": {
            "@timestamp": {
              "gte": "now-10m",
              "lte": "now"
            }
          }
        }
      ]
    }
  },
  "aggs": {
    "fields": {
      "terms": {
        "size": 100,
        "script": {
          "source": "params._source.keySet()"
        }
      }
    }
  }
}

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.