Logstash 401 Error

Elastic Stack Logstash
1

Hello,

I have followed the steps in the below doc to enable security ;

Everything seems to have worked. HOwever, when I start logstash, i get the below error :

[2020-05-24T02:16:15,655][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://localhost:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :error=>"Got response code '401' contacting Elasticsearch at URL 'http://localhost:9200/'"}

[2020-05-24T02:16:15,669][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://localhost:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :error=>"Got response code '401' contacting Elasticsearch at URL 'http://localhost:9200/'"}

[2020-05-24T02:16:15,683][WARN ][logstash.outputs.elasticsearch][main] Attempted to resurrect connection to dead ES instance, but got an error. {:url=>"http://localhost:9200/", :error_type=>LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError, :error=>"Got response code '401' contacting Elasticsearch at URL 'http://localhost:9200/'"}

I have not made any changes to logstash.yml file.

Could you please let me know what am I missing here ?

When I try to connect directly, it seems to be working.

image

2

I configured the logstash_internal user and now I see the below error during the start up of logstash :

[2020-05-24T03:48:58,345][INFO ][logstash.outputs.elasticsearch][main] Attempting to install template {:manage_template=>{"index_patterns"=>"logstash-", "version"=>60001, "settings"=>{"index.refresh_interval"=>"5s", "number_of_shards"=>1, "index.lifecycle.name"=>"logstash-policy", "index.lifecycle.rollover_alias"=>"logstash"}, "mappings"=>{"dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date"}, "@version"=>{"type"=>"keyword"}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}
warning: thread "Ruby-0-Thread-12: :1" terminated with exception (report_on_exception is true):
LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError: Got response code '403' contacting Elasticsearch at URL 'http://localhost:9200/logstash'
perform_request at /elk/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.3.3-java/lib/logstash/outputs/elasticsearch/http_client/manticore_adapter.rb:80
perform_request_to_url at /elk/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.3.3-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:332
perform_request at /elk/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.3.3-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:319
with_connection at /elk/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.3.3-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:414
perform_request at /elk/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.3.3-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:318
Pool at /elk/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.3.3-java/lib/logstash/outputs/elasticsearch/http_client/pool.rb:326
exists? at /elk/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.3.3-java/lib/logstash/outputs/elasticsearch/http_client.rb:341
rollover_alias_exists? at /elk/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.3.3-java/lib/logstash/outputs/elasticsearch/http_client.rb:359
maybe_create_rollover_alias at /elk/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.3.3-java/lib/logstash/outputs/elasticsearch/ilm.rb:91
setup_ilm at /elk/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.3.3-java/lib/logstash/outputs/elasticsearch/ilm.rb:10
setup_after_successful_connection at /elk/logstash/vendor/bundle/jruby/2.5.0/gems/logstash-output-elasticsearch-10.3.3-java/lib/logstash/outputs/elasticsearch/common.rb:54

3

Same issue was reported in : LogStash::Outputs::ElasticSearch::HttpClient::Pool::BadResponseCodeError: Got response code '403' contacting Elasticsearch at URL 'http://localhost:9200/logstash'

That post got auto closed,,,

Any suggestions please ?

4

401 is authentication error, while 403 is authorization error

your logstash_user is trying to install template in elasticsearch but it doesn’t seem to have privileges to do so. what’s your logstash_user privileges that you configure ? you will need manage_index_templates on cluster level to manage index templates.

further info on securing logstash - ES here

5

Hello

Thanks for your reply.

I have actually followed the same document.

I had created the user logstash_internal and configured that in the output section of the logstash conf file as below :

elasticsearch {
hosts => ["localhost:9200"]
user => "logstash_internal"
password => "PASSWORD"
index => "adminaccesslogsindex"
}
stdout { codec => rubydebug }
}

With the above settings, I was receiving the 403 error.

However, i changed the above setting to elastic user ( which is a super user ) and then everything was working fine.

With this, I am under an assumption that the user logstash_internal is missing some privileges to create connections to elasticsearch.

How can I know what is missing??

I had followed the steps provided at : https://www.elastic.co/guide/en/logstash/current/ls-security.html

Created the role : logstash_writer
For cluster privileges, i have added manage_index_templates and monitor .
For indices privileges, I have added write , create , delete , and create_index

Created logstash_internal user and assigned the role logstash_writer to that user.

Then created a role logstash_reader
The role has access to : read and view_index_metadata privileges for the Logstash indices

Assigned that role to logstash_internal

Yet I get 403 error.

Any comments on what could be wrong ??

6

From the Logstash error, it seems you're getting a 403 error because Logstash tries to install an index template.

Did you grant the permissions to install index templates?

7

you defined the index name as “adminaccesslogindex”. does logstash_writer has privileges to that index?

8

I have assigned logstash_reader and logstash_writer role to the user logstash_internal

Below are the screenshot of the roles configured.

image
image998×762 42.2 KB

image
image1039×731 44.3 KB

9

I have uploaded the screenshot of the roles which are assigned to the logstash_internal user.

Can you please check that and let me know if I am missing it ?

10

can you add the logstash* to the list of indices the logstash writer role has privileges to?

11

I will try it out during the weekend and let you know...

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.