Exclude fields from being indexed via dynamic templates mapping

Elastic Stack Elasticsearch
1

I have the following kind of json being indexed into ElasticSearch via Logstash:

"gameUpdate" : {
  "someField" : "something interesting",
  "otherStuff" : "...",
  "gameScores" : {
    "playerID1" : { ... },
    "playerID2" : { ... },
    "playerID3" : { ... },
    ...
  }
}

There could be a a large number of unique player IDs. I routinely run into "Could not index event to Elasticsearch. ... Limit of total fields [1000] in index [...] has been exceeded"

I have tried setting a custom template for this index as follows:

PUT _template/game-updates
{
  "index_patterns": ["game-updates-*"],
  "mappings": {
    "doc": {
      "dynamic_templates": [
        {
          "dont_index_scores": {
            "path_match": "gameUpdate.gameScores",
            "mapping": {
              "index": false
            }
          }
        }
      ]
    }
  }
}

For the path to match, I've tried "gameUpdate.gameScores" and "gameUpdate.gameScores.*" and various other variants. If I retrieve the mapping for an index created from this template, it echoes that gameScores should not be indexed. However, I still see the logstash errors.

What is the correct way to exclude these fields from the index?

2

try

"gameUpdate" : {
 "properties: {
        "gameScores": {
          "type": "object",
          "enabled": false
        },
  }
..

hope this helps

3

Is your suggestion meant for manually mapping the index, or for a dynamic mapping template? Can you please provide a more complete example which includes the

"mappings": {
  "doc": {
    "dynamic_templates": [
        ...
4

you can put that directly into the mapping, as you do not need to match anything, you are referring directly to the fields.

5

Can I mix both a dynamic mapping template and this one property specified in the mapping?

Edit: my index is auto-created every day when logstash rolls over to a new date. It's not feasible to create an explicit mapping for it. I need to use a dynamic mapping template that I can set once and have it apply to any index created which matches a certain pattern.

6

yes, you can.

7

How do I specify a mapping when my index is auto-created every day by logstash?

8

via an index template (logstash is using the exact same mechanism)

9

Thanks - this worked. I didn't know I could explicitly specify a field in the mapping (to be disabled, for example) while letting dynamic mapping still take care of the rest of the fields it encounters.