Logstash not starting and elastic complaining about old user

Elastic Stack Logstash
1

Hi

I am unable to get logstash to send data to its pipeline ever since i changed the user accounts of the cluster. I think it's trying to use an old logstash user to write to its pipeline but even if i change it back to the old user "logstash_cust_system" it wont work.

OS: Ubuntu 18.04
Elastic/Logstash/Kibana: Latest, 6.3
X-Pack is installed on trial but will have a license eventually

If there is any more information needed, please ask.

Here is the logstash log:

 Starting tcp input listener {:address=>"0.0.0.0:514", :ssl_enable=>"false"}
[2018-06-26T11:52:24,016][INFO ][logstash.inputs.beats    ] Beats inputs: Starting input listener {:address=>"0.0.0.0:5044"}
[2018-06-26T11:52:24,496][INFO ][logstash.inputs.udp      ] Starting UDP listener {:address=>"0.0.0.0:514"}
[2018-06-26T11:52:24,504][INFO ][logstash.pipeline        ] Pipeline started successfully {:pipeline_id=>"main", :thread=>"#<Thread:0x2e2eef3d@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:245 sleep>"}
[2018-06-26T11:52:24,510][INFO ][logstash.agent           ] Pipelines running {:count=>2, :running_pipelines=>[:main, :".monitoring-logstash"], :non_running_pipelines=>[]}
[2018-06-26T11:52:24,511][INFO ][logstash.inputs.metrics  ] Monitoring License OK
[2018-06-26T11:52:24,515][INFO ][org.logstash.beats.Server] Starting server on port: 5044
[2018-06-26T11:52:24,601][INFO ][logstash.inputs.udp      ] UDP listener started {:address=>"0.0.0.0:514", :receive_buffer_bytes=>"106496", :queue_size=>"2000"}
[2018-06-26T11:52:24,672][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}
[2018-06-26T11:52:26,219][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://logstash_internal:xxxxxx@10.229.1.12:9200/, :path=>"/"}
[2018-06-26T11:52:26,252][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://logstash_internal:xxxxxx@10.229.1.12:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :error=>"Got response code '401' contacting Elasticsearch at URL 'http://10.229.1.12:9200/'"}
[2018-06-26T11:52:26,252][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://logstash_internal:xxxxxx@10.229.1.13:9200/, :path=>"/"}
[2018-06-26T11:52:26,271][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://logstash_internal:xxxxxx@10.229.1.13:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :error=>"Got response code '401' contacting Elasticsearch at URL 'http://10.229.1.13:9200/'"}
[2018-06-26T11:52:31,311][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://logstash_internal:xxxxxx@10.229.1.12:9200/, :path=>"/"}
[2018-06-26T11:52:31,352][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://logstash_internal:xxxxxx@10.229.1.12:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :error=>"Got response code '401' contacting Elasticsearch at URL 'http://10.229.1.12:9200/'"}
[2018-06-26T11:52:31,352][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://logstash_internal:xxxxxx@10.229.1.13:9200/, :path=>"/"}
[2018-06-26T11:52:31,377][WARN ][logstash.outputs.elasticsearch] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://logstash_internal:xxxxxx@10.229.1.13:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :error=>"Got response code '401' contacting Elasticsearch at URL 'http://10.229.1.13:9200/'"}
[2018-06-26T11:52:32,085][FATAL][logstash.runner          ] An unexpected error occurred! {:error=>#<NoMethodError: undefined method `<' for nil:NilClass>, :backtrace=>["/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.1.1-java/lib/logstash/outputs/elasticsearch/common.rb:222:in `get_event_type'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.1.1-java/lib/logstash/outputs/elasticsearch/common.rb:47:in `event_action_tuple'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.1.1-java/lib/logstash/outputs/elasticsearch/common.rb:36:in `block in multi_receive'", "org/jruby/RubyArray.java:2486:in `map'", "/usr/share/logstash/vendor/bundle/jruby/2.3.0/gems/logstash-output-elasticsearch-9.1.1-java/lib/logstash/outputs/elasticsearch/common.rb:36:in `multi_receive'", "org/logstash/config/ir/compiler/OutputStrategyExt.java:109:in `multi_receive'", "org/logstash/config/ir/compiler/OutputDelegatorExt.java:156:in `multi_receive'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:475:in `block in output_batch'", "org/jruby/RubyHash.java:1343:in `each'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:474:in `output_batch'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:426:in `worker_loop'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:384:in `block in start_workers'"]}

And here is the elastic log

[2018-06-26T11:54:14,376][DEBUG][r.suppressed             ] path: /, params: {}
org.elasticsearch.ElasticsearchSecurityException: unable to authenticate user [logstash_cust_system] for REST request [/]
        at org.elasticsearch.xpack.core.security.support.Exceptions.authenticationError(Exceptions.java:24) ~[?:?]
        at org.elasticsearch.xpack.core.security.authc.DefaultAuthenticationFailureHandler.failedAuthentication(DefaultAuthenticationFailureHandler.java:25) ~[?:?]
        at org.elasticsearch.xpack.security.authc.AuthenticationService$AuditableRestRequest.authenticationFailed(AuthenticationService.java:580) ~[?:?]
        at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.consumeUser(AuthenticationService.java:356) ~[?:?]
        at org.elasticsearch.xpack.security.authc.AuthenticationService$Authenticator.lambda$consumeToken$14(AuthenticationService.java:294) ~[?:?]
        at org.elasticsearch.action.ActionListener$1.onResponse(ActionListener.java:60) ~[elasticsearch-
2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.