Elasticsearch output plugin in logstash not working when upgraded to version 7.1.0

Hi Team,

I am using a http_poller input plugin and some filters and an elasticsearch output (cloud instance) plugin with Logstash 6.3.0 version. All was working fine when using a ES v6.3.0 endpoint.

conf File;

input
{
	http_poller {
		urls => {
			activemq => {
				method => get
				url => "https://activemq.host.aws.amazon.in:8162/admin/xml/queues.jsp"
				user => "user"
				password => "pass"
				headers => {
					Accept => "application/json"
				}
			}
		}
		request_timeout => 60
		codec => "plain"
		schedule => { "every" => "45s" }
	}
}

filter
{
	xml
	{
		source => "message"
		target => "parsed"
	}
	
	ruby { code => 'event.set("numOfQueues", event.get("[parsed][queue]").length)' }
	
	split
	{
		field => "[parsed][queue]"
		add_field => {
			queue_name     => "%{[parsed][queue][name]}"
			queue_size     => "%{[parsed][queue][stats][0][size]}"
			consumer_count => "%{[parsed][queue][stats][0][consumerCount]}"
		}
	}
		
	mutate
	{
		convert =>{
			"queue_size"     => "integer"
			"consumer_count" => "integer"
			"numOfQueues"    => "integer"
		}
		remove_field => [ "message", "host", "parsed" ]
	}
}

output
{
	stdout
	{
		codec => rubydebug
	}
  
	elasticsearch
	{
		index => "mt_activemq_monitoring_local"
		hosts => ["https://elkhost-v7.1.0.ap-southeast-1.aws.found.io:9243"]
		user => "elastic"
		password => "password"
		#hosts => ["https://elkhost-v6.3.0.ap-southeast-1.aws.found.io:9243"]
	}
}

Output;

logstash_bin_path>logstash -f "C:\Users\Kaushik\Desktop\activemq_monitoring.conf"
[INFO ][logstash.runner          ] Starting Logstash {"logstash.version"=>"6.3.2"}
[INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
[INFO ][logstash.outputs.elasticsearch] Attempting to install template
[INFO ][logstash.inputs.http_poller] Registering http_poller Input {:type=>nil, :schedule=>{"every"=>"45s"}, :timeout=>nil}
[INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

When the endpoint in the elasticsearch plugin was changed to another one which is a version 7.1.0 version i am getting the following error and the pipeline is not starting.

Error;

[root@buildserver ~]# /usr/share/logstash/bin/logstash -f "/home/mtr_kvankayala/temp/activemq_monitoring.conf"
[INFO ] [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"6.3.1"}
[INFO ] [[main]-pipeline-manager] elasticsearch - ES Output version determined {:es_version=>7}
[INFO ] [Ruby-0-Thread-5: :1] elasticsearch - Using mapping template from {:path=>nil}
[INFO ] [Ruby-0-Thread-5: :1] elasticsearch - Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1}, "mappings"=>{"_doc"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[INFO ] [Ruby-0-Thread-5: :1] elasticsearch - Installing elasticsearch template to _template/logstash
[ERROR] [Ruby-0-Thread-5: :1] elasticsearch - Failed to install template. {:message=>"Got response code '400' contacting Elasticsearch at URL 'https://elkhost.ap-southeast-1.aws.found.io:9243/_template/logstash'", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError", :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.2.0-java/lib/logstash/outputs/elasticsearch/http_client/manticore_adapter.rb:80:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.2.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:291:in `perform_request_to_url'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.2.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:278:in `block in perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.2.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:373:in `with_connection'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.2.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:277:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.2.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:285:in `block in Pool'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.2.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:348:in `template_put'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.2.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:86:in `template_install'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.2.0-java/lib/logstash/outputs/elasticsearch/template_manager.rb:21:in `install'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.2.0-java/lib/logstash/outputs/elasticsearch/template_manager.rb:9:in `install_template'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.2.0-java/lib/logstash/outputs/elasticsearch/common.rb:118:in `install_template'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.2.0-java/lib/logstash/outputs/elasticsearch/common.rb:49:in `block in install_template_after_successful_connection'"]}

Note : I am running logstash as a service and i have two files placed in the /etc/logstash/conf.d directory

Could you kindly help me what is the thing that needs to be considered to resolve the error?

Regards

Kaushik

Could you test the connectivity to your elasticsearch with curl?
eg:
curl -s -X GET --user elastic:password 'https://elkhost-v6.3.0.ap-southeast-1.aws.found.io:9243'

@pastechecker i could do that and i am very well able to receive the response;

$ curl -s -X GET --user elastic:password 'https://elkhost.ap-southeast-1.aws.found.io:9243'
{
  "name" : "instance-0000000000",
  "cluster_name" : "elkhost",
  "cluster_uuid" : "XGRQk_hkj802gfjkjj1Lwa7g",
  "version" : {
    "number" : "7.1.0",
    "build_flavor" : "default",
    "build_type" : "tar",
    "build_hash" : "606a173",
    "build_date" : "2019-05-16T00:43:15.323135Z",
    "build_snapshot" : false,
    "lucene_version" : "8.0.0",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

I am wondering why the same conf file that works with 6.x does not work with 7.y
I have checked the breakchange document and could not find any conflict with the plugins that i am using.
I am wondering why the template installation error is occurring.

Can you advise further?

If it was the authentication problem you would get 401 response code.
How about the write permissions for the user? Does the role that is assigned to the user for ingestion have cluster privileges: manage_index_templates and manage_ilm ?

Hi @pastechecker, yes i have indeed provided the credentials of admin here. Also when i tried to download the Logstash 7.1.0 and tried to execute the same conf file with the ES endpoint v7.1.0 i had not faced any error. But when i run the same conf file with Logstash 6.3.2 i am getting that error.

So how do i go about resolving the error if i want to keep using the Logstash 6.3.2?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.