SOLVED: Logstash issue with shield - "action [indices:data/write/bulk] is unauthorized for user [logstash]"


(Clement Ros) #1


I am about to install shield on my cluster and when I want to configure logstash to communicate with ES i have the following problem :

  • I followed the documentation and when i did this configuration the authentication didn't work:

elasticsearch {
user => "myuser"
password => "mypassword"

  • Here is my list of users

logstash : logstash
kibana4-server : kibana4_server
rosc : kibana4
admin : admin

I didn't touch my roles.yml file.

i have the following error in the Logstash log file:

error_message=>"[403] {"error":{"root_cause":[{"type":"security_exception","reason":"action [indices:data/write/bulk] is unauthorized for user [logstash]"}]

Anyone have an idea ?

(Jay Modi) #2

what are the indices that your logstash instance configured to write into?

(Michal) #3


I You have in your roles.yml "kibana" index, please replace to "kibana*".
Should works :slight_smile:

(Clement Ros) #4

Hi jay,

My logstash is configured to write into indices "%{[@metadata][beat]}-%{+YYYY.MM.dd}", "prtsys-%{+YYYY.MM.dd}" and "logstash-%{+YYYY.MM.dd}"

Yesterday I try to assign the admin role to my logstash user, but it did not remove the error, but I now see the data appear in Kibana.

Here is my admin roles :

`# All operations on all indices
  cluster: all
      privileges: all`

(Clement Ros) #5

Hi Michal,

Do you speak of the '.kibana' ?

(Jay Modi) #6

Hi Clement,

That is very odd that the admin role did not allow the user to write into the indices. Do you think you could enable auditing and grab the access denied entry? This should contain more information about the failure to help diagnose it.


(Clement Ros) #7

Here are my logs in access.log file:

[2016-04-27 13:42:13,757] [uat-node09] [transport] [access_granted] origin_type=[rest], origin_address=[], principal=[logstash], action=[indices:data/write/bulk[s]], indices=[logstash-2016.04.27,logstash-2016.04.27,logstash-2016.04.27,logstash-2016.04.27,logstash-2016.04.27,logstash-2016.04.27,logstash-2016.04.27,logstash-2016.04.27,logstash-2016.04.27,logstash-2016.04.27,logstash-2016.04.27,logstash-2016.04.27,logstash-2016.04.27,logstash-2016.04.27,logstash-2016.04.27,logstash-2016.04.27,logstash-2016.04.27,logstash-2016.04.27,logstash-2016.04.27,logstash-2016.04.27,logstash-2016.04.27,logstash-2016.04.27,logstash-2016.04.27,logstash-2016.04.27,logstash-2016.04.27]

[2016-04-27 13:42:13,756] [uat-node09] [transport] [access_granted] origin_type=[rest], origin_address=[], principal=[logstash], action=[indices:data/write/bulk]

[2016-04-27 13:42:13,303] [uat-node09] [transport] [access_granted] origin_type=[rest], origin_address=[], principal=[logstash], action=[indices:data/write/bulk[s]], indices=[topbeat-2016.04.27]

(Jay Modi) #8

So there are no access denied entries and you still see that exception regarding write not being allowed for your logstash user?

(Clement Ros) #9

Yes this is it,

My file logstash.log continues to fill the following logs :

{:timestamp=>"2016-04-27T14:25:52.328000+0200", :message=>"Attempted to send a bulk request to Elasticsearch configured at '["", "", ""]', but an error occurred and it failed! Are you sure you can reach elasticsearch from this machine using the configuration provided?", :client_config=>{:hosts=>["", "", ""] ...


{:timestamp=>"2016-04-27T14:25:52.328000+0200", :message=>"[401] {"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user [logstash] for REST request [/_bulk]","header":{"WWW-Authenticate":"Basic realm=\"shield\""}}],"type":"security_exception","reason":"unable to authenticate user [logstash] for REST request [/_bulk]","header":{"WWW-Authenticate":"Basic realm=\"shield\""}},"status":401}", ...

(Jay Modi) #10

This appears to be a different issue. It appears as though the password may not be correct for the logstash user on this instance of logstash? I am thinking that you should see some authentication failure audit logs for that as well.

(Clement Ros) #11

I test to generate this error on my other instances without success.

So i copied the logstash conf file from a good instance and it works !!!!

Thanks Jay

(Clement Ros) #12

This morning the errors reappeared, and they appeared on each instance.

So i try to change the password of my logstash users on each elasticsearch instance to be sure to have the same password as the one entered in the logstash conf file.

And the error still appears.

I try to restart my Elasticsearch and logstash instances but nothing changed.

And all i have in my access.log file, is access_granted messages... :sob:

(Jay Modi) #13

This is very odd. What versions of logstash and elasticsearch/shield are you using? The logstash user is a file based user (or esusers) correct? If so, can you validate that the user's password is the same on each node and that the roles files are in sync on each node?

(Clement Ros) #14

It's my fault :sweat: , logstash the user was not created on some of my two knots , I had forgotten to recreate it after deleting . Sorry about that. I will wait until tomorrow to see if no error is returned.

Thanks for your help.

(Jay Modi) #15

Hopefully it does not return. You may want to consider moving to 2.3 so you can use the users API and avoid dealing with keeping these users in sync manually.

(system) #16