Should I use `keyword` in `term` queries?

Elastic Stack Elasticsearch
1

I parse a file with filebeat and send it to elasticsearch where I parse it with an ingest node into fields. The problem I have is that I want to use term queries to match the text of one of those fields.

Until 2 days ago, I could use:

"term": {
  "name.of.field.keyword": {
    "value": "exact_value_here"
  }
}

and I would get back the results.

But now, the above query leaves out some newer documents. If I remove .keyword from the query, I get the newer documents but I miss all the old ones.

The mapping in the index was created as:

          "event": {
            "type": "text",
            "fields": {
              "keyword": {
                "type": "keyword",
                "ignore_above": 256
              }
            }
          },

but in newer indexes it is created as:

          "event": {
            "type": "keyword",
            "ignore_above": 1024
          },

My questions:

  • What is the best way to match exactly the content of a string field?
  • How come the same field changed mapping? The same version of both filebeat and elasticsearch was used. How can I make sure that all these filebeat fields are of type keyword and not text?
2

You introduced a non backward compatible mapping change.

The correct way to cope with them might be to create a field alias on the new indices.
The alias keyword, under event, will point to event so you'll be able to query on event.keyword.

3

The issue is that I do not control the indexes directly. The elasticsearch ingest node creates the fields on the fly in the filebeat index (that in turn has been created using the default index template that comes with filebeat). I have never so far needed to mess with the indexes themselves other than to insert the index template.

4

Just to clarify: you were not using the index templates and now you've started using them, is it correct?

5

No, I used them from the beginning. But of course there are no mappings for the custom fields that filebeat creates.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.