Exclude fields from being indexed via dynamic templates mapping

Elastic Stack Elasticsearch
1

I have the following kind of json being indexed into ElasticSearch via Logstash:

"gameUpdate" : {
  "someField" : "something interesting",
  "otherStuff" : "...",
  "gameScores" : {
    "playerID1" : { ... },
    "playerID2" : { ... },
    "playerID3" : { ... },
    ...
  }
}

There could be a a large number of unique player IDs. I routinely run into "Could not index event to Elasticsearch. ... Limit of total fields [1000] in index [...] has been exceeded"

I have tried setting a custom template for this index as follows:

PUT _template/game-updates
{
  "index_patterns": ["game-updates-*"],
  "mappings": {
    "doc": {
      "dynamic_templates": [
        {
          "dont_index_scores": {
            "path_match": "gameUpdate.gameScores",
            "mapping": {
              "index": false
            }
          }
        }
      ]
    }
  }
}

For the path to match, I've tried "gameUpdate.gameScores" and "gameUpdate.gameScores.*" and various other variants. If I retrieve the mapping for an index created from this template, it echoes that gameScores should not be indexed. However, I still see the logstash errors.

What is the correct way to exclude these fields from the index?

2

try

"gameUpdate" : {
 "properties: {
        "gameScores": {
          "type": "object",
          "enabled": false
        },
  }
..

hope this helps

3

Is your suggestion meant for manually mapping the index, or for a dynamic mapping template? Can you please provide a more complete example which includes the

"mappings": {
  "doc": {
    "dynamic_templates": [
        ...
4

you can put that directly into the mapping, as you do not need to match anything, you are referring directly to the fields.

5

Can I mix both a dynamic mapping template and this one property specified in the mapping?

Edit: my index is auto-created every day when logstash rolls over to a new date. It's not feasible to create an explicit mapping for it. I need to use a dynamic mapping template that I can set once and have it apply to any index created which matches a certain pattern.

6

yes, you can.

7

How do I specify a mapping when my index is auto-created every day by logstash?

8

via an index template (logstash is using the exact same mechanism)

9

Thanks - this worked. I didn't know I could explicitly specify a field in the mapping (to be disabled, for example) while letting dynamic mapping still take care of the rest of the fields it encounters.

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.