Logstash -> Elasticsearch index template

Elastic Stack Elasticsearch
1

I am running Logstash and a three-node Elasticsearch cluster in docker containers. In the Elasticsearch output plugin of Logstash, I'm attempting to use a custom index template with the following lines in logstash.conf:

manage_template => true
template_name => "my-logs"
template => "/usr/share/logstash/template.json"

When I run this, I get a 'failed to install template' error with code 400 from Elasticsearch, which I've interpreted to mean there's something wrong with my template, but from looking at the examples I can't see anything wrong. I've also tried making sure Elasticsearch is definitely up and running before starting Logstash.

{
    "index_patterns" : ["my-logs*"],
    "priority" : 1,
    "template": {
    	"settings" : {
    		"number_of_shards" : 1,
    		"number_of_replicas" : 1
    	},
    	"mappings": {
    		"properties": {
    			"geoip":{
    				"dynamic": true,
    				"type" : "object",
    				"properties": {
    					"location": {
    						"type" : "geo_point"
    					}
    				}
    			}
    		}
    	}
    }
}

Edit: I forgot to add that the the logs are first coming from Filebeat, although that shouldn't be relevant.

2

Which version of ES are you running?

3

7.8.0

4

If you run Logstash with --debug do you get more information about that HTTP 400?

5

Changing logging level to debug didn't give me any more information, but here is the full error message:

Failed to install template. {:message=>"Got response code '400' contacting Elasticsearch at URL 'http://es01:9200/_template/my-logs'", :class=>"LogStash::Outputs::Elasticsearch::HttpClient::Pool::BadResponseCodeError", :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.5.1-java/lib/logstash/outputs/elasticsearch/http_client/manticore_adapter.rb:80:in perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.5.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:332:in perform_request_to_url'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.5.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:319:in block in perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.5.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:414:in with_connection'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.5.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:318:in perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.5.1-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:326:in block in Pool'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.5.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:352:in template_put'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.5.1-java/lib/logstash/outputs/elasticsearch/http_client.rb:86:in template_install'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.5.1-java/lib/logstash/outputs/elasticsearch/template_manager.rb:28:in install'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.5.1-java/lib/logstash/outputs/elasticsearch/template_manager.rb:16:in install_template'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.5.1-java/lib/logstash/outputs/elasticsearch/common.rb:205:in install_template'", "/usr/share/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.5.1-java/lib/logstash/outputs/elasticsearch/common.rb:49:in block in setup_after_successful_connection'"]}

6

Oh, I know why...

Logstash doesn't yet support the new index templates, but only the legacy index templates.

If you change your template to the following one, it will work:

{
    "index_patterns" : ["my-logs*"],
    "order" : 1,
	"settings" : {
		"number_of_shards" : 1,
		"number_of_replicas" : 1
	},
	"mappings": {
		"properties": {
			"geoip":{
				"dynamic": true,
				"type" : "object",
				"properties": {
					"location": {
						"type" : "geo_point"
					}
				}
			}
		}
	}
}

I've opened a new issue to track this.

7

Thank you, this worked! Or at least got rid of the error. Will legacy templates continue to be supported in the update?

8

Cool, glad it helped!

If the issue will be handled, I suppose the elasticsearch output plugin should continue working with both legacy and new templates for the foreseeable future.

9

Out of curiosity, what's the advantage of the new template compared to the old one?

10

The biggest advantage is that now index templates will be composable out of component templates.

The main reason they did this (I believe) was to help with the redesign of the new Ingest Manager and Data Streams, where each data set will be able to provide its own component templates. This and probably many other reasons, too.

1 Like
11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.