Logstash user in Xpack

Elastic Stack Logstash
1

there are 2 configuration files for our logstash:

  • config/logstash.yml
  • pipeline/logstash.conf

Now I want to know which roles and which user(s) we minimal need to define to get data to elasticsearch and to see it in kibana.

must the logstash.yml user be the same user als the user in logstash.conf?
Which roles do we need? The build_in logstash_system user had only the logstash_system role which does not seem to be enough for indices:

Got a bad response code from server, but this code is not considered retryable. Request will be dropped {:code=>403, :response_body=>"{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"action [indices:data/write/bulk] is unauthorized for user [logstash_system]\"}],\"type\":\"security_exception\",\"reason\":\"action [indices:data/write/bulk] is unauthorized for user [logstash_system]\"},\"status\":403}"}

This role does not provide access to the logstash indices and is not suitable for use within a Logstash pipeline.

2

The documentation for setting up a logstash with X-Pack security is here:

You need to create your own user (the docs use logstash_internal with role logstash_writer) and grant it the specific permissions that are needed for your use of logstash.

1 Like
4

I did the configuration like on the URL. It works. It's visible in kibana. Log's are send. But the logs of logstash itself are a bit 'strange'.

Every time I perform a manual curl to trigger logs. It's going through logstash so all fine. But when there isn't really happening anything it's showing some forbidden log. I don't know what logstash tries to do. Maybe some health check?

So I did nothing. Than I did a curl (10 times on my apache container). Logs are from logstash to Elastic search. Fine. Then I wait again a bit. Then I curl 4 times. So it works but the logstash logging isn't what I want! Thanks

 tion\",\"reason\":\"action [cluster:admin/xpack/monitoring/bulk] is unauthorized for user [logstash_internal]\"},\"status\":403}"}
[2017-08-03T20:20:58,516][ERROR][logstash.outputs.elasticsearch] Got a bad response code from server, but this code is not considered retryable. Request will be dropped {:code=>403, :response_body=>"{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"action [cluster:admin/xpack/monitoring/bulk] is unauthorized for user [logstash_internal]\"}],\"type\":\"security_exception\",\"reason\":\"action [cluster:admin/xpack/monitoring/bulk] is unauthorized for user [logstash_internal]\"},\"status\":403}"}
[2017-08-03T20:21:08,535][ERROR][logstash.outputs.elasticsearch] Got a bad response code from server, but this code is not considered retryable. Request will be dropped {:code=>403, :response_body=>"{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"action [cluster:admin/xpack/monitoring/bulk] is unauthorized for user [logstash_internal]\"}],\"type\":\"security_exception\",\"reason\":\"action [cluster:admin/xpack/monitoring/bulk] is unauthorized for user [logstash_internal]\"},\"status\":403}"}
2017-08-03T20:21:12.487Z  172.17.0.1 - - [03/Aug/2017:20:21:12 +0000] "GET / HTTP/1.1" 200 45
2017-08-03T20:21:12.516Z  172.17.0.1 - - [03/Aug/2017:20:21:12 +0000] "GET / HTTP/1.1" 200 45
2017-08-03T20:21:12.533Z  172.17.0.1 - - [03/Aug/2017:20:21:12 +0000] "GET / HTTP/1.1" 200 45
2017-08-03T20:21:12.546Z  172.17.0.1 - - [03/Aug/2017:20:21:12 +0000] "GET / HTTP/1.1" 200 45
2017-08-03T20:21:12.556Z  172.17.0.1 - - [03/Aug/2017:20:21:12 +0000] "GET / HTTP/1.1" 200 45
2017-08-03T20:21:12.572Z  172.17.0.1 - - [03/Aug/2017:20:21:12 +0000] "GET / HTTP/1.1" 200 45
2017-08-03T20:21:12.589Z  172.17.0.1 - - [03/Aug/2017:20:21:12 +0000] "GET / HTTP/1.1" 200 45
2017-08-03T20:21:12.598Z  172.17.0.1 - - [03/Aug/2017:20:21:12 +0000] "GET / HTTP/1.1" 200 45
2017-08-03T20:21:12.611Z  172.17.0.1 - - [03/Aug/2017:20:21:12 +0000] "GET / HTTP/1.1" 200 45
2017-08-03T20:21:12.630Z  172.17.0.1 - - [03/Aug/2017:20:21:12 +0000] "GET / HTTP/1.1" 200 45
[2017-08-03T20:21:18,532][ERROR][logstash.outputs.elasticsearch] Got a bad response code from server, but this code is not considered retryable. Request will be dropped {:code=>403, :response_body=>"{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"action [cluster:admin/xpack/monitoring/bulk] is unauthorized for user [logstash_internal]\"}],\"type\":\"security_exception\",\"reason\":\"action [cluster:admin/xpack/monitoring/bulk] is unauthorized for user [logstash_internal]\"},\"status\":403}"}
[2017-08-03T20:21:28,542][ERROR][logstash.outputs.elasticsearch] Got a bad response code from server, but this code is not considered retryable. Request will be dropped {:code=>403, :response_body=>"{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"action [cluster:admin/xpack/monitoring/bulk] is unauthorized for user [logstash_internal]\"}],\"type\":\"security_exception\",\"reason\":\"action [cluster:admin/xpack/monitoring/bulk] is unauthorized for user [logstash_internal]\"},\"status\":403}"}
[2017-08-03T20:21:38,540][ERROR][logstash.outputs.elasticsearch] Got a bad response code from server, but this code is not considered retryable. Request will be dropped {:code=>403, :response_body=>"{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"action [cluster:admin/xpack/monitoring/bulk] is unauthorized for user [logstash_internal]\"}],\"type\":\"security_exception\",\"reason\":\"action [cluster:admin/xpack/monitoring/bulk] is unauthorized for user [logstash_internal]\"},\"status\":403}"}
[2017-08-03T20:21:49,181][ERROR][logstash.outputs.elasticsearch] Got a bad response code from server, but this code is not considered retryable. Request will be dropped {:code=>403, :response_body=>"{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"action [cluster:admin/xpack/monitoring/bulk] is unauthorized for user [logstash_internal]\"}],\"type\":\"security_exception\",\"reason\":\"action [cluster:admin/xpack/monitoring/bulk] is unauthorized for user [logstash_internal]\"},\"status\":403}"}
[2017-08-03T20:21:58,552][ERROR][logstash.outputs.elasticsearch] Got a bad response code from server, but this code is not considered retryable. Request will be dropped {:code=>403, :response_body=>"{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"action [cluster:admin/xpack/monitoring/bulk] is unauthorized for user [logstash_internal]\"}],\"type\":\"security_exception\",\"reason\":\"action [cluster:admin/xpack/monitoring/bulk] is unauthorized for user [logstash_internal]\"},\"status\":403}"}
[2017-08-03T20:22:08,553][ERROR][logstash.outputs.elasticsearch] Got a bad response code from server, but this code is not considered retryable. Request will be dropped {:code=>403, :response_body=>"{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"action [cluster:admin/xpack/monitoring/bulk] is unauthorized for user [logstash_internal]\"}],\"type\":\"security_exception\",\"reason\":\"action [cluster:admin/xpack/monitoring/bulk] is unauthorized for user [logstash_internal]\"},\"status\":403}"}
2017-08-03T20:22:11.375Z  172.17.0.1 - - [03/Aug/2017:20:22:11 +0000] "GET / HTTP/1.1" 200 45
2017-08-03T20:22:11.398Z  172.17.0.1 - - [03/Aug/2017:20:22:11 +0000] "GET / HTTP/1.1" 200 45
2017-08-03T20:22:11.456Z  172.17.0.1 - - [03/Aug/2017:20:22:11 +0000] "GET / HTTP/1.1" 200 45
2017-08-03T20:22:11.456Z  172.17.0.1 - - [03/Aug/2017:20:22:11 +0000] "GET / HTTP/1.1" 200 45
[2017-08-03T20:22:18,566][ERROR][logstash.outputs.elasticsearch] Got a bad response code from server, but this code is not considered retryable. Request will be dropped {:code=>403, :response_body=>"{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"action [cluster:admin/xpack/monitoring/bulk] is unauthorized for user [logstash_internal]\"}],\"type\":\"security_exception\",\"reason\":\"action [cluster:admin/xpack/monitoring/bulk] is unauthorized for user [logstash_internal]\"},\"status\":403}"}

Same issue as: Logstash security_exception, can't write to ES after installing X-pack I think

5

adding monitor_role fixed it.

1 Like
6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.