Defining index mappings on ElasticSearch

Elastic Stack Elasticsearch
,
1

Hi there,

This is my first post here, so I apologise in advance if I'm not too clear or if my question has been asked already.

I have ElasticSearch running in a Docker cluster, I use it to search through a ASP.NET core API that queries ES.
I would like to define the index mappings for ES at cluster creation, running a job in my docker-compose that uses a .NET console app to put the requests on ES to define the mappings.

I have seen that using templates should do the job for me, but I can't find any examples for best practices in NEST or ElasticSearch-NET so I thought I'd ask here.

My idea would be to simply define a template that is applied to any new index that is created. Then I run an external job that seeds ES and any new index will have the same mapping applied to it whenever a new document needs to go in a new index.
I also want to define an alias that is common to all the new indexes.

The mappings I want to define are as below:

    {
    	"settings": {
    	  "analysis": {
    		"filter": {
    		  "autocomplete_filter": {
    			"type": "edge_ngram",
    			"min_gram": 1,
    			"max_gram": 20
    		  }
    		},
    		"analyzer": {
    		  "autocomplete": {
    			"type": "custom",
    			"tokenizer": "standard",
    			"filter": [
    			  "lowercase",
    			  "autocomplete_filter"
    			]
    		  }
    		}
    	  }
    	},
    	"mappings": {
    	  "dynamic": false,
    	  "properties": {
    		"Name": {
    		  "type": "keyword"
    		},
    		"Cusip": {
    		  "type": "keyword"
    		},
    		"ISIN": {
    		  "type": "keyword"
    		},
    		"Ticker": {
    		  "type": "keyword"
    		},
    		"suggest": {
    		  "type": "completion",
    		  "analyzer": "autocomplete",
    		  "search_analyzer": "standard"
    		}
    	  }
    	}
    }

And for the alias I need to use something like this:

    {
              "actions" : [
                  { "add" : { "index" : "{index}", "alias" : "product" } }
              ]
          }

How do I package it together into a template? Is using a template the right way to do this? How do I ensure that all new indexes have these settings?

Thanks in advance for your help
Simon

2

At the top of my templates I have:

{
  "index_patterns" : [ "test_data_index_hlq-*" ],
  "template" : {
    "settings" : {
      "number_of_shards" : 1
    },
    "mappings" : {
      "properties" : {

        "@timestamp" : {
          "type" : "date"
        },

That seems to be enough to tell it to use the template for any index that matches the pattern. The trailing - on the pattern may be significant - or maybe you just have to be consistent with kibana and transforms.

3

In the end is solved this using the LowLevel Elasticsearch-net client. I have a config file that contains the JSON request for the template:

{"index_patterns": [
    "*"
  ],
  "settings": {
    "analysis": {
      "filter": {
        "autocomplete_filter": {
          "type": "edge_ngram",
          "min_gram": 1,
          "max_gram": 20
        }
      },
      "analyzer": {
        "autocomplete": {
          "type": "custom",
          "tokenizer": "standard",
          "filter": [
            "lowercase",
            "autocomplete_filter"
          ]
        }
      }
    }
  },
  "mappings": {
    "dynamic": false,
    "properties": {
      "Name": {
        "type": "keyword"
      },
      "field1": {
        "type": "keyword"
      },
      "field2": {
        "type": "keyword"
      },
      "field3": {
        "type": "keyword"
      },
      "suggest": {
        "type": "completion",
        "analyzer": "autocomplete",
        "search_analyzer": "standard"
      }
    }
  },
  "aliases" : {
    "product" : {}
  }
}

And I send the request for the template mapping like this:

// Send a PUT request containing the template
var postMapping =
        _client.LowLevel.DoRequest<StringResponse>(HttpMethod.PUT, "_template/product_template", PostData.String(template.ToString()));
4

I still have to solve how to ensure that only new indices get my template, and not the metrics or logs indices. If I define the "index_pattern" to be "*" that will apply to all new indices. Is there a way to specify that I don't want to apply the template to indices starting with "."?

The examples for index_pattern are not very helpful, somehow we expect indices that share the same template to start with the same letter, like "t*" etc. In my use case I have a list of more than 30 types of products that I want to share the same settings, and I don't want to add every product type to the index_pattern list. On top of that, I want to be able to use the same template for all the new products that might be added in the future, since I know that my ES cluster will always contain objects that belong to that category. Is there a way to achieve this, without changing the index name of all products?

So far the only solution I've found is to prepend a string to the index names so that I have them be like "q-abc", "q-xyz" etc. The index pattern matching seems to only allow for wildcard matching, is there a plan to implement something like RegEx or more advanced pattern matching solutions?

@Russ_Cam

5

You can use the low level client API method, and return a high level response if using the low level client exposed on the high level client

var client = new ElasticClient();

var json = @"{""index_patterns"": [
	""*""
  ],
  ""settings"": {
		""analysis"": {
			""filter"": {
				""autocomplete_filter"": {
					""type"": ""edge_ngram"",
          ""min_gram"": 1,
          ""max_gram"": 20
		
		}
			},
      ""analyzer"": {
				""autocomplete"": {
					""type"": ""custom"",
          ""tokenizer"": ""standard"",
          ""filter"": [

			""lowercase"",
            ""autocomplete_filter""
		          ]
        }
			}
		}
	},
  ""mappings"": {
		""dynamic"": false,
    ""properties"": {
			""Name"": {
				""type"": ""keyword""
	  
	  },
      ""field1"": {
				""type"": ""keyword""

	  },
      ""field2"": {
				""type"": ""keyword""

	  },
      ""field3"": {
				""type"": ""keyword""

	  },
      ""suggest"": {
				""type"": ""completion"",
        ""analyzer"": ""autocomplete"",
        ""search_analyzer"": ""standard""

	  }
		}
	},
  ""aliases"" : {
		""product"" : { }
	}
}
";

var response = client.LowLevel.Indices.PutTemplateForAll<PutIndexTemplateResponse>("product_template", json);

I think index patterns are intended to work on the basis that you know and control the names of indices created, so that index templates apply only to a subset of indices. Configuring an index template with the * pattern indicates that you want to apply a template to all indices, but that doesn't sound like what you want to do. The order parameter can be used to define the order in which multiple templates apply, which may help,

6

Thanks so much for your reply!

Your suggestion to use:
client.LowLevel.Indices.PutTemplateForAll<PutIndexTemplateResponse>("product_template", json);
seems a lot better, to be honest I couldn't find any reference to this method in the .NET client after version 1.x, maybe the docs could be updated to make it more clear?

Regarding the pattern matching issue, I don't think the order would solve it because it would still be applied to all indices, and that causes a fatal error when spinning up Kibana because the index mappings for the metrics/logs indices are misconfigured.

I'll follow @warkolm's advice here and open a feature request for a "not match" index pattern matching.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.