I did the configuration like on the URL. It works. It's visible in kibana. Log's are send. But the logs of logstash itself are a bit 'strange'.
Every time I perform a manual curl to trigger logs. It's going through logstash so all fine. But when there isn't really happening anything it's showing some forbidden log. I don't know what logstash tries to do. Maybe some health check?
So I did nothing. Than I did a curl (10 times on my apache container). Logs are from logstash to Elastic search. Fine. Then I wait again a bit. Then I curl 4 times. So it works but the logstash logging isn't what I want! Thanks
tion\",\"reason\":\"action [cluster:admin/xpack/monitoring/bulk] is unauthorized for user [logstash_internal]\"},\"status\":403}"}
[2017-08-03T20:20:58,516][ERROR][logstash.outputs.elasticsearch] Got a bad response code from server, but this code is not considered retryable. Request will be dropped {:code=>403, :response_body=>"{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"action [cluster:admin/xpack/monitoring/bulk] is unauthorized for user [logstash_internal]\"}],\"type\":\"security_exception\",\"reason\":\"action [cluster:admin/xpack/monitoring/bulk] is unauthorized for user [logstash_internal]\"},\"status\":403}"}
[2017-08-03T20:21:08,535][ERROR][logstash.outputs.elasticsearch] Got a bad response code from server, but this code is not considered retryable. Request will be dropped {:code=>403, :response_body=>"{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"action [cluster:admin/xpack/monitoring/bulk] is unauthorized for user [logstash_internal]\"}],\"type\":\"security_exception\",\"reason\":\"action [cluster:admin/xpack/monitoring/bulk] is unauthorized for user [logstash_internal]\"},\"status\":403}"}
2017-08-03T20:21:12.487Z 172.17.0.1 - - [03/Aug/2017:20:21:12 +0000] "GET / HTTP/1.1" 200 45
2017-08-03T20:21:12.516Z 172.17.0.1 - - [03/Aug/2017:20:21:12 +0000] "GET / HTTP/1.1" 200 45
2017-08-03T20:21:12.533Z 172.17.0.1 - - [03/Aug/2017:20:21:12 +0000] "GET / HTTP/1.1" 200 45
2017-08-03T20:21:12.546Z 172.17.0.1 - - [03/Aug/2017:20:21:12 +0000] "GET / HTTP/1.1" 200 45
2017-08-03T20:21:12.556Z 172.17.0.1 - - [03/Aug/2017:20:21:12 +0000] "GET / HTTP/1.1" 200 45
2017-08-03T20:21:12.572Z 172.17.0.1 - - [03/Aug/2017:20:21:12 +0000] "GET / HTTP/1.1" 200 45
2017-08-03T20:21:12.589Z 172.17.0.1 - - [03/Aug/2017:20:21:12 +0000] "GET / HTTP/1.1" 200 45
2017-08-03T20:21:12.598Z 172.17.0.1 - - [03/Aug/2017:20:21:12 +0000] "GET / HTTP/1.1" 200 45
2017-08-03T20:21:12.611Z 172.17.0.1 - - [03/Aug/2017:20:21:12 +0000] "GET / HTTP/1.1" 200 45
2017-08-03T20:21:12.630Z 172.17.0.1 - - [03/Aug/2017:20:21:12 +0000] "GET / HTTP/1.1" 200 45
[2017-08-03T20:21:18,532][ERROR][logstash.outputs.elasticsearch] Got a bad response code from server, but this code is not considered retryable. Request will be dropped {:code=>403, :response_body=>"{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"action [cluster:admin/xpack/monitoring/bulk] is unauthorized for user [logstash_internal]\"}],\"type\":\"security_exception\",\"reason\":\"action [cluster:admin/xpack/monitoring/bulk] is unauthorized for user [logstash_internal]\"},\"status\":403}"}
[2017-08-03T20:21:28,542][ERROR][logstash.outputs.elasticsearch] Got a bad response code from server, but this code is not considered retryable. Request will be dropped {:code=>403, :response_body=>"{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"action [cluster:admin/xpack/monitoring/bulk] is unauthorized for user [logstash_internal]\"}],\"type\":\"security_exception\",\"reason\":\"action [cluster:admin/xpack/monitoring/bulk] is unauthorized for user [logstash_internal]\"},\"status\":403}"}
[2017-08-03T20:21:38,540][ERROR][logstash.outputs.elasticsearch] Got a bad response code from server, but this code is not considered retryable. Request will be dropped {:code=>403, :response_body=>"{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"action [cluster:admin/xpack/monitoring/bulk] is unauthorized for user [logstash_internal]\"}],\"type\":\"security_exception\",\"reason\":\"action [cluster:admin/xpack/monitoring/bulk] is unauthorized for user [logstash_internal]\"},\"status\":403}"}
[2017-08-03T20:21:49,181][ERROR][logstash.outputs.elasticsearch] Got a bad response code from server, but this code is not considered retryable. Request will be dropped {:code=>403, :response_body=>"{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"action [cluster:admin/xpack/monitoring/bulk] is unauthorized for user [logstash_internal]\"}],\"type\":\"security_exception\",\"reason\":\"action [cluster:admin/xpack/monitoring/bulk] is unauthorized for user [logstash_internal]\"},\"status\":403}"}
[2017-08-03T20:21:58,552][ERROR][logstash.outputs.elasticsearch] Got a bad response code from server, but this code is not considered retryable. Request will be dropped {:code=>403, :response_body=>"{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"action [cluster:admin/xpack/monitoring/bulk] is unauthorized for user [logstash_internal]\"}],\"type\":\"security_exception\",\"reason\":\"action [cluster:admin/xpack/monitoring/bulk] is unauthorized for user [logstash_internal]\"},\"status\":403}"}
[2017-08-03T20:22:08,553][ERROR][logstash.outputs.elasticsearch] Got a bad response code from server, but this code is not considered retryable. Request will be dropped {:code=>403, :response_body=>"{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"action [cluster:admin/xpack/monitoring/bulk] is unauthorized for user [logstash_internal]\"}],\"type\":\"security_exception\",\"reason\":\"action [cluster:admin/xpack/monitoring/bulk] is unauthorized for user [logstash_internal]\"},\"status\":403}"}
2017-08-03T20:22:11.375Z 172.17.0.1 - - [03/Aug/2017:20:22:11 +0000] "GET / HTTP/1.1" 200 45
2017-08-03T20:22:11.398Z 172.17.0.1 - - [03/Aug/2017:20:22:11 +0000] "GET / HTTP/1.1" 200 45
2017-08-03T20:22:11.456Z 172.17.0.1 - - [03/Aug/2017:20:22:11 +0000] "GET / HTTP/1.1" 200 45
2017-08-03T20:22:11.456Z 172.17.0.1 - - [03/Aug/2017:20:22:11 +0000] "GET / HTTP/1.1" 200 45
[2017-08-03T20:22:18,566][ERROR][logstash.outputs.elasticsearch] Got a bad response code from server, but this code is not considered retryable. Request will be dropped {:code=>403, :response_body=>"{\"error\":{\"root_cause\":[{\"type\":\"security_exception\",\"reason\":\"action [cluster:admin/xpack/monitoring/bulk] is unauthorized for user [logstash_internal]\"}],\"type\":\"security_exception\",\"reason\":\"action [cluster:admin/xpack/monitoring/bulk] is unauthorized for user [logstash_internal]\"},\"status\":403}"}
Same issue as: Logstash security_exception, can't write to ES after installing X-pack I think