Applying My First Index Template

Hi everyone,

I’m trying to apply my first ElasticSearch Index Template, and am having some issues. I think I’m both a little confused about the process of wedding a Template to an Index, and about the exact syntax of the template itself.

The mission: My ElasticSearch receives network data from a Logstash server. Within the raw data are IP addresses, which ES currently interprets to be text strings. I need an Index Template to transform those strings into proper “ip” data structures.

My plan was this:

  • Reconfigure Logstash to send data into a new index called “MyIndex.” But do not start Logstash yet.
  • Create a new Index Template called “my_template” in ES. Make sure template “my_template” specifies that it is to be used with “MyIndex” (See below)
  • Start Logstash

I’m not clear how ES understands which templates get applied to what Indices, but I think that’s done with a line in the template itself.

Speaking of, here’s the Index Template I cobbled together:

curl -X PUT "localhost:9200/_template/my_template" -H 'Content-Type: application/json' -d'
{
    "template": "MyIndex",
    "order": 1,
        "settings": {
            "index": {
            "refresh_interval": "5s"
            }
        },
    "mappings": {
        "default": {
            "_all": {
                "norms": false,
                "enabled": true
                },
                "properties": {
                    "Packet.L3.Src": { "type": "ip"},
                    "Packet.L3.Dst": { "type": "ip"}
            }
        }
    }
}'

As you can see, all I really want it to do is interpret the data fields “Packet.L3.Src” and “Packet.L3.Dst” as IP addresses. Also, note that first line within the outer brackets: "template": "MyIndex". I assume this is how ES understands that this template is to be applied to Index “MyIndex.”

As a safety check, I ran the above template through a JSON validator. (here) Everything looked good on that front.

So I should be ready to rock-n-roll now. However, when I cut-n-paste the template into ES, I get the following error (I’ve included newlines for better readability):

{"error":
	{"root_cause":
		[{"type":"mapper_parsing_exception",
		"reason":"Root mapping definition has unsupported parameters:  
		[default : {_all={norms=false, enabled=true}, 
		properties={Packet.L3.Src={type=ip}, Packet.L3.Dst={type=ip}}}]"}],
		"type":"mapper_parsing_exception",
		"reason":"Failed to parse mapping [_doc]: 
		Root mapping definition has unsupported parameters:  
		[default : 
			{_all={norms=false, enabled=true}, 
			properties={Packet.L3.Src={type=ip}, Packet.L3.Dst={type=ip}}}]",
			"caused_by":
			{"type":"mapper_parsing_exception",
			"reason":"Root mapping definition has unsupported parameters:  
			[default : {_all={norms=false, enabled=true}, 
			properties={Packet.L3.Src={type=ip}, Packet.L3.Dst={type=ip}}}]"
}
}
,"status":400 } 

I’m not sure what this means, but when I see things like “parsing exception,” I assume I have a syntax error that is throwing off the parsing of my template. Like, maybe I have a comma at the end of a line where I shouldn’t… or vice versa…? I don’t know.

FULL DISCLOSURE: I’ve been working on this issue in the ES Forum, and posted an earlier part of my issues was posted here.

Anyway, I’m hoping someone can point out the error of my ways. Any advice is appreciated!

Does anyone have any thoughts? My development project is dead in the water until I can push past this issue. Thanks...!

There's a few things going on here. You did not specify what version of Elasticsearch you are using, but I'm going to assume you are on version 7. There have been a few changes to how index templates and mappings are defined in recent versions. Some of the syntax you are using does not work in 7 any more:

  • Instead of template you need to set index_patterns. By the way, this is how Elasticsearch matches a template to an index. Whenever you create a new index, Elasticsearch will try to find any index template with an index-patterns pattern that matches the name of that new index.
  • Document types have gone away. You no longer need to (and no longer can) provide default in the mapping.
  • _all went away too.

Applying these changes to your index template, the correct request to create your index template would be:

PUT /_template/my_template
{
  "index_patterns": "MyIndex",
  "order": 1,
  "settings": {
    "index": {
      "refresh_interval": "5s"
    }
  },
  "mappings": {
    "properties": {
      "Packet.L3.Src": {
        "type": "ip"
      },
      "Packet.L3.Dst": {
        "type": "ip"
      }
    }
  }
}

Thanks Abdon, I missed your note because of the holiday in the US. I'll be in the office tomorrow and will deep dive on your notes. Thank you for replying! I hadn't realized I'd gotten so many wires crossed...

Much appreciated,
-Pete

Many thanks to Abdon! Yes, you were entirely correct. I am on ES version 7.3.0, and hadn't realized I was cross-pollinating solutions from different ES versions. I must have found the core of my solution in an old Google search, then added in elements from more modern documentation. Good catch, you've saved my project!

Much appreciated,
-P

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.