Hi Team,
I am using a http_poller input plugin and some filters and an elasticsearch output (cloud instance) plugin with Logstash 6.3.0 version. All was working fine when using a ES v6.3.0 endpoint.
conf File;
input
{
http_poller {
urls => {
activemq => {
method => get
url => "https://activemq.host.aws.amazon.in:8162/admin/xml/queues.jsp"
user => "user"
password => "pass"
headers => {
Accept => "application/json"
}
}
}
request_timeout => 60
codec => "plain"
schedule => { "every" => "45s" }
}
}
filter
{
xml
{
source => "message"
target => "parsed"
}
ruby { code => 'event.set("numOfQueues", event.get("[parsed][queue]").length)' }
split
{
field => "[parsed][queue]"
add_field => {
queue_name => "%{[parsed][queue][name]}"
queue_size => "%{[parsed][queue][stats][0][size]}"
consumer_count => "%{[parsed][queue][stats][0][consumerCount]}"
}
}
mutate
{
convert =>{
"queue_size" => "integer"
"consumer_count" => "integer"
"numOfQueues" => "integer"
}
remove_field => [ "message", "host", "parsed" ]
}
}
output
{
stdout
{
codec => rubydebug
}
elasticsearch
{
index => "mt_activemq_monitoring_local"
hosts => ["https://elkhost-v7.1.0.ap-southeast-1.aws.found.io:9243"]
user => "elastic"
password => "password"
#hosts => ["https://elkhost-v6.3.0.ap-southeast-1.aws.found.io:9243"]
}
}
Output;
logstash_bin_path>logstash -f "C:\Users\Kaushik\Desktop\activemq_monitoring.conf"
[INFO ][logstash.runner ] Starting Logstash {"logstash.version"=>"6.3.2"}
[INFO ][logstash.outputs.elasticsearch] ES Output version determined {:es_version=>6}
[INFO ][logstash.outputs.elasticsearch] Attempting to install template
[INFO ][logstash.inputs.http_poller] Registering http_poller Input {:type=>nil, :schedule=>{"every"=>"45s"}, :timeout=>nil}
[INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
When the endpoint in the elasticsearch plugin was changed to another one which is a version 7.1.0 version i am getting the following error and the pipeline is not starting.
Error;
[root@buildserver ~]# /usr/share/logstash/bin/logstash -f "/home/mtr_kvankayala/temp/activemq_monitoring.conf"
[INFO ] [LogStash::Runner] runner - Starting Logstash {"logstash.version"=>"6.3.1"}
[INFO ] [[main]-pipeline-manager] elasticsearch - ES Output version determined {:es_version=>7}
[INFO ] [Ruby-0-Thread-5: :1] elasticsearch - Using mapping template from {:path=>nil}
[INFO ] [Ruby-0-Thread-5: :1] elasticsearch - Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1}, "mappings"=>{"_doc"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[INFO ] [Ruby-0-Thread-5: :1] elasticsearch - Installing elasticsearch template to _template/logstash
[ERROR] [Ruby-0-Thread-5: :1] elasticsearch - Failed to install template. {:message=>"Got response code '400' contacting Elasticsearch at URL 'https://elkhost.ap-southeast-1.aws.found.io:9243/_template/logstash'", :class=>"LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError", :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.2.0-java/lib/logstash/outputs/elasticsearch/http_client/manticore_adapter.rb:80:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.2.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:291:in `perform_request_to_url'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.2.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:278:in `block in perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.2.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:373:in `with_connection'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.2.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:277:in `perform_request'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.2.0-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:285:in `block in Pool'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.2.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:348:in `template_put'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.2.0-java/lib/logstash/outputs/elasticsearch/http_client.rb:86:in `template_install'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.2.0-java/lib/logstash/outputs/elasticsearch/template_manager.rb:21:in `install'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.2.0-java/lib/logstash/outputs/elasticsearch/template_manager.rb:9:in `install_template'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.2.0-java/lib/logstash/outputs/elasticsearch/common.rb:118:in `install_template'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.2.0-java/lib/logstash/outputs/elasticsearch/common.rb:49:in `block in install_template_after_successful_connection'"]}
Note : I am running logstash as a service and i have two files placed in the /etc/logstash/conf.d directory
Could you kindly help me what is the thing that needs to be considered to resolve the error?
Regards
Kaushik