X-pack on logstash


(Shashank Jain) #1

Hi All ,

I have installed X-pack on logstash but in step it is mentioned to add user and pwd in logstash.yml .

but i am using pipelines which have configuration files under conf.d ,
so where shall i add x-pack related settings .

Update Logstash to use the new password for the built-in logstash_system user, which you set up along with the other built-in users when you installed X-Pack on Elasticsearch. You must configure the xpack.monitoring.elasticsearch.password setting in the logstash.yml configuration file with the new password for the logstash_system user.

xpack.monitoring.elasticsearch.username: logstash_system
xpack.monitoring.elasticsearch.password: logstashpassword


(Mark Walkom) #2

Those two go in logstash.yml


(Shashank Jain) #3

Yup i have updated these details in logstash.yml also below settings also ,

xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.url: ["http://node01:9200","http://node02:9200","http://node03:9200"]
xpack.monitoring.elasticsearch.username: logstash_system
xpack.monitoring.elasticsearch.password: XXXXXXXXXX

but i am seeing below exception while starting logstash ,

[2018-01-05T00:00:04,329][WARN ][logstash.outputs.elasticsearch] You are using a deprecated config setting "document_type" set in
elasticsearch. Deprecated settings will continue to work, but are scheduled for removal from logstash in the future. Document type
s are being deprecated in Elasticsearch 6.0, and removed entirely in 7.0. You should avoid this feature If you have any questions
about this, please visit the #logstash channel on freenode irc. {:name=>"document_type", :plugin=><LogStash::Outputs::ElasticSearc
h hosts=>[http://vielk01dsy.dsone.3ds.com:9200, http://vielk02dsy.dsone.3ds.com:9200, http://vpkiba1dsy.dsone.3ds.com:9200], bulk_
path=>"/_xpack/monitoring/_bulk?system_id=logstash&system_api_version=2&interval=1s", manage_template=>false, document_type=>"%{[@
metadata][document_type]}", sniffing=>false, user=>"logstash_system", password=>, id=>"e2ff6781ffaafd4b5fe25b53c84ba2f89
d28fd61c871ada00b153a1f3a3fbe7d", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_582aa62d-0178-4f91-b184-1e70cb03
f84a", enable_metric=>true, charset=>"UTF-8">, workers=>1, template_name=>"logstash", template_overwrite=>false, doc_as_upsert=>fa
lse, script_type=>"inline", script_lang=>"painless", script_var_name=>"event", scripted_upsert=>false, retry_initial_interval=>2,
retry_max_interval=>64, retry_on_conflict=>1, action=>"index", ssl_certificate_verification=>true, sniffing_delay=>5, timeout=>60,
pool_max=>1000, pool_max_per_route=>100, resurrect_delay=>5, validate_after_inactivity=>10000, http_compression=>false>}
[2018-01-05T00:00:04,577][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=

[http://logstash_system:xxxxxx@vielk01dsy.dsone.3ds.com:9200/, http://logstash_system:xxxxxx@vielk02dsy.dsone.3ds.com:9200/, http
://logstash_system:xxxxxx@vpkiba1dsy.dsone.3ds.com:9200/]}}

[2018-01-05T00:00:04,729][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the type event field won't be
used to determine the document _type {:es_version=>6}

[2018-01-05T00:00:04,751][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>".monitoring-logstash", "pipeline.wor
kers"=>1, "pipeline.batch.size"=>2, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>2, :thread=>"#<Thread:0x74e3ffe2@/usr/shar
e/logstash/logstash-core/lib/logstash/pipeline.rb:245 run>"}

[2018-01-05T00:00:06,371][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error.
{:url=>"http://vielk01dsy.dsone.3ds.com:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError,
:error=>"Got response code '401' contacting Elasticsearch at URL 'http://vielk01dsy.dsone.3ds.com:9200/'"}

[2018-01-05T00:00:06,379][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error.
{:url=>"http://vielk02dsy.dsone.3ds.com:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError,
:error=>"Got response code '401' contacting Elasticsearch at URL 'http://vielk02dsy.dsone.3ds.com:9200/'"}

[2018-01-05T00:00:06,392][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error.
{:url=>"http://vpkiba1dsy.dsone.3ds.com:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError,
:error=>"Got response code '401' contacting Elasticsearch at URL 'http://vpkiba1dsy.dsone.3ds.com:9200/'"}

[2018-01-05T00:00:06,392][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}


(Shashank Jain) #4

exception continued ,

[2018-01-05T00:00:06,394][ERROR][logstash.outputs.elasticsearch] Failed to install template. {:message=>"Template file '' could not be found!",
:class=>"ArgumentError",
:backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.2-java/
lib/logstash/outputs/elasticsearch/template_manager.rb:31:
in read_template_file'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.2-java/ lib/logstash/outputs/elasticsearch/template_manager.rb:17: inget_template'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/
logstash-output-elasticsearch-9.0.2-java/lib/logstash/outputs/elasticsearch/template_manager.rb:7
:in install_template'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.2-java/lib/logstash/outputs/ elasticsearch/common.rb:57:ininstall_template'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.2-java/lib/
logstash/outputs/elasticsearch/common.rb:26:in register'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator_strategies/shared.rb:9:inregister'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator.rb:43:in register'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:343:inregister_plugin'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:354:in
block in register_plugins'", "org/jruby/RubyArray.java:1734:ineach'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:354:in
register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:743: inmaybe_setup_out_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:364
:in start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:288: inrun'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:248:in `block in start'"]}

what template it is talking about and trying to load .
aslo my ES nodes are running fine but i am not able to see any data in indexes .


(Mark Walkom) #5

401 indicates an authentication problem, so I would check the user against the cluster manually.


(system) #6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.