Elasticsearch permissions required by Filebeat

[alastairs (Alastair Smith)]

I'm looking to secure my Elastic Cloud deployment by creating additional users (and roles where necessary) on the system. I'm not comfortable using the root elastic username and password for writing logs from Filebeat, and so I'd like to create a new user specifically for Filebeat ingestion. What permissions does Filebeat need in Elasticsearch to function correctly? I haven't been able to find this information via Google or in the Filebeat docs.

[adrisr (Adrian Serrano)]

Hi,

Have a look at Securing Filebeat section of the docs.

[alastairs (Alastair Smith)]

Thanks @adrisr. Please can you confirm whether this works with Cloud authentication (in conjunction with the cloud ID) also? It seems as though it should, but I'm seeing authentication failures as a result of changing this.

Additionally, from step 4 of that page of the documentation, I don't see a beats_system user in my deployment, so I guess there's nothing to do there. Perhaps the docs need updating for Elastic Cloud?

[adrisr (Adrian Serrano)]

It worked for me, leaving cloud.id as is, and changing cloud.auth to the new user:

cloud.auth: filebeat_internal:YOUR_PASSWORD

About step 4 you are right, I don't see any beats_system user either, I will raise the issue

[alastairs (Alastair Smith)]

Great, thanks. I tried with a different password (no symbols) and it worked fine. Not sure which symbol caused the problem there, but I assume there are some forbidden ones of which I fell foul.

I'm seeing a lot of these messages in my filebeat logs now:

2018-11-02T18:03:21.874Z        ERROR   pipeline/output.go:121  Failed to publish 
events: 403 Forbidden: {"error":{"root_cause":[{"type":"security_exception",
"reason":"action [cluster:admin/xpack/monitoring/bulk] is unauthorized for user
 [filebeat]"}],"type":"security_exception","reason":"action 
[cluster:admin/xpack/monitoring/bulk] is unauthorized for user [filebeat]"},
"status":403}

I can't see to which permission this action maps; any ideas which I've missed, or indeed what's causing it? I'm using Kubernetes autodiscover, and these are the modules I have enabled:

Enabled modules/filesets: apache2 (access), traefik (access), redis (log, slowlog),  (), system (auth, syslog)

Not sure where that blank one has come from

[alastairs (Alastair Smith)]

Aha, it's the beats_system role it needs. Found that in the Monitoring Filebeat docs

[adrisr (Adrian Serrano)]

Thanks for your feedback! I'm glad you sorted it out.

I've created an issue to improve the documentation on this.