Can't connect to ElasticSearch after X-pack install

dyl · February 8, 2018, 2:07pm

Hello. I just follow steps to configure x-pack with Logstash ([Installing X-pack In Logsatsh])(Installing X-Pack in Logstash | Logstash Reference [6.2] | Elastic)

But after this, my logstash pipeline doesn't work anymore ..

Here it is my logstash.yml :

path.data: /var/lib/logstash
path.logs: /var/log/logstash
xpack.monitoring.elasticsearch.username: logstash_system
xpack.monitoring.elasticsearch.password: mypass

Here it is my output pipeline :

output {
elasticsearch {
hosts => ["localhost:9200"]
user => logstash_system
password => mypass
}
stdout { codec => rubydebug }
}

and here the mistakes that I have :

[INFO ] 2018-02-08 15:01:10.290 [Ruby-0-Thread-10: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:228] licensereader - Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://localhost:9200/, :path=>"/"}
[WARN ] 2018-02-08 15:01:10.297 [Ruby-0-Thread-10: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.0.2-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:228] licensereader - Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://localhost:9200/", :error_type=>LogStash::Outputs::Elasticsearch::HttpClient::Pool::BadResponseCodeError, :error=>"Got response code '401' contacting Elasticsearch at URL 'http://localhost:9200/'"}
[ERROR] 2018-02-08 15:01:11.436 [Ruby-0-Thread-1: /usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:22] agent - Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of #, {, } at line 29, column 23 (byte 680) after output {\n elasticsearch { \n hosts => ["localhost:9200"]\n user => logstash_system\n password => Azerty5", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:42:in `compile_imperative'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:50:in `compile_graph'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:12:in `block in compile_sources'", "org/jruby/RubyArray.java:2486:in `map'", "/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:11:in `compile_sources'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:51:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:169:in `initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:40:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:315:in `block in converge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:in `with_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:312:in `block in converge_state'", "org/jruby/RubyArray.java:1734:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:299:in `converge_state'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:166:in `block in converge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:141:in `with_pipelines'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:164:in `converge_state_and_update'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:105:in `block in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/interval.rb:18:in `interval'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:94:in `execute'", "/usr/share/logstash/logstash-core/lib/logstash/runner.rb:348:in `block in execute'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/stud-0.0.23/lib/stud/task.rb:24:in `block in initialize'"]}

What cause this issue ?

thx !

jpcarey · February 8, 2018, 2:32pm

401 unauthorized. Did you setup the user / password for writing the monitoring data? This is done in the logstash.yml file

It is also complaining about an invalid config, at line 29 column 23. I couldn’t spot the issue from the truncated config provided, you need to fully check that the config is formed correctly.

Also I see you configured the elasticsearch output in your pipeline to use logstash_system. This user is meant for writing monitoring data, and does not have the necessary permissions to create other data. These docs describe setting up an appropriate logstash_writer user:
https://www.elastic.co/guide/en/logstash/current/ls-security.html

dyl · February 8, 2018, 2:56pm

Ok, I just created the user "logstash_internal" with the role "logstash_writer" exactly as indicated in your link (by changing the password of course :))

I modified logstash.yml which now becomes :

path.data: /var/lib/logstash
path.logs: /var/log/logstash
xpack.monitoring.elasticsearch.username: logstash_internal
xpack.monitoring.elasticsearch.password: newpass

And I also changed my pipeline which now becomes:

input {
udp {
port => 514
type => syslog
}
}

FILTER SECTION

filter {
if [type] == "syslog" {
grok {
match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:[%{POSINT:syslog_pid}])?: %{GREEDYDATA:syslog_message}" }
add_field => [ "received_at", "%{@timestamp}" ]
add_field => [ "received_from", "%{host}" ]
}
date {
  match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
}
}

OUTPUT SECTION

output {
elasticsearch {
hosts => ["localhost:9200"]
user => "logstash_internal"
password => "newpass"
}
stdout { codec => rubydebug }
}