X-pack on logstash

Elastic Stack Logstash
1

Hi All ,

I have installed X-pack on logstash but in step it is mentioned to add user and pwd in logstash.yml .

but i am using pipelines which have configuration files under conf.d ,
so where shall i add x-pack related settings .

Update Logstash to use the new password for the built-in logstash_system user, which you set up along with the other built-in users when you installed X-Pack on Elasticsearch. You must configure the xpack.monitoring.elasticsearch.password setting in the logstash.yml configuration file with the new password for the logstash_system user.

xpack.monitoring.elasticsearch.username: logstash_system
xpack.monitoring.elasticsearch.password: logstashpassword

2

Those two go in logstash.yml

3

Yup i have updated these details in logstash.yml also below settings also ,

xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.url: ["http://node01:9200","http://node02:9200","http://node03:9200"]
xpack.monitoring.elasticsearch.username: logstash_system
xpack.monitoring.elasticsearch.password: XXXXXXXXXX

but i am seeing below exception while starting logstash ,

[2018-01-05T00:00:04,329][WARN ][logstash.outputs.elasticsearch] You are using a deprecated config setting "document_type" set in
elasticsearch. Deprecated settings will continue to work, but are scheduled for removal from logstash in the future. Document type
s are being deprecated in Elasticsearch 6.0, and removed entirely in 7.0. You should avoid this feature If you have any questions
about this, please visit the logstash channel on freenode irc. {:name=>"document_type", :plugin=><LogStash::Outputs::ElasticSearc
h hosts=>[http://vielk01dsy.dsone.3ds.com:9200, http://vielk02dsy.dsone.3ds.com:9200, http://vpkiba1dsy.dsone.3ds.com:9200], bulk_
path=>"/_xpack/monitoring/_bulk?system_id=logstash&system_api_version=2&interval=1s", manage_template=>false, document_type=>"%{[@
metadata][document_type]}", sniffing=>false, user=>"logstash_system", password=>, id=>"e2ff6781ffaafd4b5fe25b53c84ba2f89
d28fd61c871ada00b153a1f3a3fbe7d", enable_metric=>true, codec=><LogStash::Codecs::Plain id=>"plain_582aa62d-0178-4f91-b184-1e70cb03
f84a", enable_metric=>true, charset=>"UTF-8">, workers=>1, template_name=>"logstash", template_overwrite=>false, doc_as_upsert=>fa
lse, script_type=>"inline", script_lang=>"painless", script_var_name=>"event", scripted_upsert=>false, retry_initial_interval=>2,
retry_max_interval=>64, retry_on_conflict=>1, action=>"index", ssl_certificate_verification=>true, sniffing_delay=>5, timeout=>60,
pool_max=>1000, pool_max_per_route=>100, resurrect_delay=>5, validate_after_inactivity=>10000, http_compression=>false>}
[2018-01-05T00:00:04,577][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>, :added=

[http://logstash_system:xxxxxx@vielk01dsy.dsone.3ds.com:9200/, http://logstash_system:xxxxxx@vielk02dsy.dsone.3ds.com:9200/, http
://logstash_system:xxxxxx@vpkiba1dsy.dsone.3ds.com:9200/]}}

[2018-01-05T00:00:04,729][WARN ][logstash.outputs.elasticsearch] Detected a 6.x and above cluster: the type event field won't be
used to determine the document _type {:es_version=>6}

[2018-01-05T00:00:04,751][INFO ][logstash.pipeline ] Starting pipeline {:pipeline_id=>".monitoring-logstash", "pipeline.wor
kers"=>1, "pipeline.batch.size"=>2, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>2, :thread=>"#<Thread:0x74e3ffe2@/usr/shar
e/logstash/logstash-core/lib/logstash/pipeline.rb:245 run>"}

[2018-01-05T00:00:06,371][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error.
{:url=>"http://vielk01dsy.dsone.3ds.com:9200/", :error_type=>LogStash::Outputs::Elasticsearch::HttpClient::Pool::BadResponseCodeError,
:error=>"Got response code '401' contacting Elasticsearch at URL 'http://vielk01dsy.dsone.3ds.com:9200/'"}

[2018-01-05T00:00:06,379][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error.
{:url=>"http://vielk02dsy.dsone.3ds.com:9200/", :error_type=>LogStash::Outputs::Elasticsearch::HttpClient::Pool::BadResponseCodeError,
:error=>"Got response code '401' contacting Elasticsearch at URL 'http://vielk02dsy.dsone.3ds.com:9200/'"}

[2018-01-05T00:00:06,392][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error.
{:url=>"http://vpkiba1dsy.dsone.3ds.com:9200/", :error_type=>LogStash::Outputs::Elasticsearch::HttpClient::Pool::BadResponseCodeError,
:error=>"Got response code '401' contacting Elasticsearch at URL 'http://vpkiba1dsy.dsone.3ds.com:9200/'"}

[2018-01-05T00:00:06,392][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}

4

exception continued ,

[2018-01-05T00:00:06,394][ERROR][logstash.outputs.elasticsearch] Failed to install template. {:message=>"Template file '' could not be found!",
:class=>"ArgumentError",
:backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.2-java/
lib/logstash/outputs/elasticsearch/template_manager.rb:31:
in read_template_file'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.2-java/ lib/logstash/outputs/elasticsearch/template_manager.rb:17: inget_template'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/
logstash-output-elasticsearch-9.0.2-java/lib/logstash/outputs/elasticsearch/template_manager.rb:7
:in install_template'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.2-java/lib/logstash/outputs/ elasticsearch/common.rb:57:ininstall_template'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.2-java/lib/
logstash/outputs/elasticsearch/common.rb:26:in register'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator_strategies/shared.rb:9:inregister'", "/usr/share/logstash/logstash-core/lib/logstash/output_delegator.rb:43:in register'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:343:inregister_plugin'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:354:in
block in register_plugins'", "org/jruby/RubyArray.java:1734:ineach'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:354:in
register_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:743: inmaybe_setup_out_plugins'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:364
:in start_workers'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:288: inrun'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:248:in `block in start'"]}

what template it is talking about and trying to load .
aslo my ES nodes are running fine but i am not able to see any data in indexes .

5

401 indicates an authentication problem, so I would check the user against the cluster manually.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.