Index template not applying to indices

Elastic Stack Elasticsearch
1

Hi there

We're using Filebeat 6.4.2 to scrape our logs from Kubernetes (with Docker) and are shipping our logs directly to Elastic Cloud, running 6.5.0. I loaded a custom index template into Elasticsearch to add a .raw field to our logs, as follows (this is a copy-paste from Serilog's Elasticsearch sink):

GET /_template/serilog-template
{
  "serilog-template" : {
    "order" : 0,
    "index_patterns" : [
      "filebeat-*"
    ],
    "settings" : {
      "index" : {
        "refresh_interval" : "5s"
      }
    },
    "mappings" : {
      "_default_" : {
        "dynamic_templates" : [
          {
            "numerics_in_fields" : {
              "path_match" : """fields\.[\d+]$""",
              "match_pattern" : "regex",
              "mapping" : {
                "type" : "text",
                "index" : true,
                "norms" : false
              }
            }
          },
          {
            "string_fields" : {
              "match" : "*",
              "match_mapping_type" : "string",
              "mapping" : {
                "type" : "text",
                "index" : true,
                "norms" : false,
                "fields" : {
                  "raw" : {
                    "type" : "keyword",
                    "index" : true,
                    "ignore_above" : 256
                  }
                }
              }
            }
          }
        ],
        "properties" : {
          "message" : {
            "type" : "text",
            "index" : "true"
          },
          "exceptions" : {
            "type" : "nested",
            "properties" : {
              "Depth" : {
                "type" : "integer"
              },
              "RemoteStackIndex" : {
                "type" : "integer"
              },
              "HResult" : {
                "type" : "integer"
              },
              "StackTraceString" : {
                "type" : "text",
                "index" : "true"
              },
              "RemoteStackTraceString" : {
                "type" : "text",
                "index" : "true"
              },
              "ExceptionMessage" : {
                "type" : "object",
                "properties" : {
                  "MemberType" : {
                    "type" : "integer"
                  }
                }
              }
            }
          }
        }
      }
    },
    "aliases" : { }
  }
}

In addition to this template, Filebeat has also installed its default template, which specifies "order": 1. I understand from the docs that in this scenario the custom template will be applied before the Filebeat template.

However, I do not see any of the .raw fields in my index mapping, or in the posted data. It seems almost as though the index pattern is being ignored entirely?

We also have the same template applied to a custom index to which we submit log events directly (i.e., without Filebeat in the picture), and this does include .raw fields in the index pattern and in the data, so my best guess is that the interplay with Filebeat is causing us problems here.

2

That is technically correct, but the important part of the docs here is "...with lower order being applied first and higher orders overriding them".

1 Like
3

Thanks @Igor_Motov! So am I correct in thinking, then, that although my custom template is applied first, the Filebeat templates are overriding it to the extent that the .raw fields are being stripped out again?

4

Once index is created, the templates have no effect on this index. So, the combined effect of all templates that were applied in the order that they were applied are in the mappings and settings of the index that was created. So, you need to take a look at the dynamic templates of one of the index that was created to see what the combined effect was.

5

The default mapping is deprecated

6

Interesting, thanks @saif, that's good to know. AFAICT the .raw fields are appearing ok in the non-Filebeat indices, but it looks like that's something to update nonetheless.

7

Ok, I've updated the index template to have order 2 (so applies after Filebeat) and map to doc rather than _default_. I've used the update by query API to reindex the data in today's index, and I've refreshed the index pattern in Kibana. Here is (what I think is) the relevant portion of the dynamic templates applied to today's index:

{
  // Created by Filebeat index template
  "strings_as_keyword" : {
    "match_mapping_type" : "string",
    "mapping" : {
      "ignore_above" : 1024,
      "type" : "keyword"
    }
  }
},
{
  // Created by custom index template
  "numerics_in_fields" : {
    "path_match" : """fields\.[\d+]$""",
    "match_pattern" : "regex",
    "mapping" : {
      "index" : true,
      "norms" : false,
      "type" : "text"
    }
  }
},
{
  // Created by custom index template
  "string_fields" : {
    "match" : "*",
    "match_mapping_type" : "string",
    "mapping" : {
      "fields" : {
        "raw" : {
          "ignore_above" : 256,
          "index" : true,
          "type" : "keyword"
        }
      },
      "index" : true,
      "norms" : false,
      "type" : "text"
    }
  }
}

I feel as though there's a conflict here between the strings_as_keyword dynamic template from Filebeat and the string_fields dynamic template from my custom index template.

(Needless to say, still no .raw fields in my index.)

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.