SOLVED: Logstash issue with shield - "action [indices:data/write/bulk] is unauthorized for user [logstash]"

Elastic Stack Logstash
1

Hi,

I am about to install shield on my cluster and when I want to configure logstash to communicate with ES i have the following problem :

  • I followed the documentation and when i did this configuration the authentication didn't work:

elasticsearch {
...
user => "myuser"
password => "mypassword"
}

  • Here is my list of users

logstash : logstash
kibana4-server : kibana4_server
rosc : kibana4
admin : admin

I didn't touch my roles.yml file.

i have the following error in the Logstash log file:

error_message=>"[403] {"error":{"root_cause":[{"type":"security_exception","reason":"action [indices:data/write/bulk] is unauthorized for user [logstash]"}]

Anyone have an idea ?

2

what are the indices that your logstash instance configured to write into?

3

Hi,

I You have in your roles.yml "kibana" index, please replace to "kibana*".
Should works :slight_smile:

4

Hi jay,

My logstash is configured to write into indices "%{[@metadata][beat]}-%{+YYYY.MM.dd}", "prtsys-%{+YYYY.MM.dd}" and "logstash-%{+YYYY.MM.dd}"

Yesterday I try to assign the admin role to my logstash user, but it did not remove the error, but I now see the data appear in Kibana.

Here is my admin roles :

`# All operations on all indices
admin:
  cluster: all
  indices:
    '*':
      privileges: all`
5

Hi Michal,

Do you speak of the '.kibana' ?

6

Hi Clement,

That is very odd that the admin role did not allow the user to write into the indices. Do you think you could enable auditing and grab the access denied entry? This should contain more information about the failure to help diagnose it.

-Jay

7

Here are my logs in access.log file:

[2016-04-27 13:42:13,757] [uat-node09] [transport] [access_granted] origin_type=[rest], origin_address=[10.1.103.1], principal=[logstash], action=[indices:data/write/bulk[s]], indices=[logstash-2016.04.27,logstash-2016.04.27,logstash-2016.04.27,logstash-2016.04.27,logstash-2016.04.27,logstash-2016.04.27,logstash-2016.04.27,logstash-2016.04.27,logstash-2016.04.27,logstash-2016.04.27,logstash-2016.04.27,logstash-2016.04.27,logstash-2016.04.27,logstash-2016.04.27,logstash-2016.04.27,logstash-2016.04.27,logstash-2016.04.27,logstash-2016.04.27,logstash-2016.04.27,logstash-2016.04.27,logstash-2016.04.27,logstash-2016.04.27,logstash-2016.04.27,logstash-2016.04.27,logstash-2016.04.27]

[2016-04-27 13:42:13,756] [uat-node09] [transport] [access_granted] origin_type=[rest], origin_address=[10.1.103.1], principal=[logstash], action=[indices:data/write/bulk]

[2016-04-27 13:42:13,303] [uat-node09] [transport] [access_granted] origin_type=[rest], origin_address=[10.1.103.1], principal=[logstash], action=[indices:data/write/bulk[s]], indices=[topbeat-2016.04.27]

8

So there are no access denied entries and you still see that exception regarding write not being allowed for your logstash user?

9

Yes this is it,

My file logstash.log continues to fill the following logs :

{:timestamp=>"2016-04-27T14:25:52.328000+0200", :message=>"Attempted to send a bulk request to Elasticsearch configured at '["http://10.1.101.1:9200/\", "http://10.1.102.1:9200/\", "http://10.1.103.1:9200/\"]', but an error occurred and it failed! Are you sure you can reach elasticsearch from this machine using the configuration provided?", :client_config=>{:hosts=>["http://10.1.101.1:9200/", "http://10.1.102.1:9200/", "http://10.1.103.1:9200/"] ...

and

{:timestamp=>"2016-04-27T14:25:52.328000+0200", :message=>"[401] {"error":{"root_cause":[{"type":"security_exception","reason":"unable to authenticate user [logstash] for REST request [/_bulk]","header":{"WWW-Authenticate":"Basic realm=\"shield\""}}],"type":"security_exception","reason":"unable to authenticate user [logstash] for REST request [/_bulk]","header":{"WWW-Authenticate":"Basic realm=\"shield\""}},"status":401}", ...

10

This appears to be a different issue. It appears as though the password may not be correct for the logstash user on this instance of logstash? I am thinking that you should see some authentication failure audit logs for that as well.

11

I test to generate this error on my other instances without success.

So i copied the logstash conf file from a good instance and it works !!!!

Thanks Jay

1 Like
12

This morning the errors reappeared, and they appeared on each instance.

So i try to change the password of my logstash users on each elasticsearch instance to be sure to have the same password as the one entered in the logstash conf file.

And the error still appears.

I try to restart my Elasticsearch and logstash instances but nothing changed.

And all i have in my access.log file, is access_granted messages... :sob:

13

This is very odd. What versions of logstash and elasticsearch/shield are you using? The logstash user is a file based user (or esusers) correct? If so, can you validate that the user's password is the same on each node and that the roles files are in sync on each node?

14

It's my fault :sweat: , logstash the user was not created on some of my two knots , I had forgotten to recreate it after deleting . Sorry about that. I will wait until tomorrow to see if no error is returned.

Thanks for your help.

15

Hopefully it does not return. You may want to consider moving to 2.3 so you can use the users API and avoid dealing with keeping these users in sync manually.

1 Like