Filebeat does not work while using cloud.id and cloud.auth

[newKibanaUser]

Hello -
I am trying to use cloud.id and cloud.auth in my file beat configuration file. But that seems not working.

This Works perfectly without using cloud.id and cloud_auth.

output.elasticsearch:
hosts: ["https://4d90d9c2814c429xxxxxxxxxx:99999"]
protocol: https
ssl: null
ignoreversion: true
username: "xxxxxxx"
password: "yyyyyyy"

This does not work while using cloud.id and cloud_auth, getting error like below.

401 Unauthorized: {"error":{"root_cause":[{"type":"security_exception","reason":"action [cluster:monitor/main] requires authentication","header":{"WWW-Authenticate":["Bearer realm="securit.......

output.elasticsearch:
hosts: ["https://4d90d9c2814c429xxxxxxxxxx:99999"]
ssl: null
ignorversion: true
cloud.id: "temps:am9uxxxxxxxxxxxxxxxxxxxxxx"
cloud.auth: "xxxxxxxx:yyyyyyyyy"

Can someone help?

Thank you.

[grumo35]

Hi,
You need to configure the permissions of your user

Here is my logstash_writer :

[newKibanaUser]

Thank you.

We are using ECE 2.4 deployed on our internal cloud.

Where or how to get to this screen?

BTW, I am not using Logstash in my filebeat.yml. Filebeat (on local instance in cloud) --> ElasticSearch (on cloud).

[grumo35]

In Kibana Users / Roles creating a new role,

if you're not using logstash this should work the same way i hope

[newKibanaUser]

Hi -

I don't see that screen in my Kibana.

I have assigned all the available roles existed for my user under Management -> Security -> Users and still filebeat fails while using cloud.id and cloud.auth and my user.

[newKibanaUser]

Didn't dig deep enough.

So, I created a new role with the cluster and index privileges as you have specified. (And, i don't see monitor_transform privileges on my dropdown).

I assigned this new role to the user and still filebeat fails.

2020-01-15T14:10:32.434-0500 ERROR pipeline/output.go:100 Failed to connect to backoff(elasticsearch(https://4d90d9c2814c429d9ce4a416921ca611.xxxxxxxx-test:xxxx)): 401 Unauthorized: {"error":{"root_cause":[{"type":"security_exception","reason":"action [cluster:monitor/main] requires authentication","header":{"WWW-Authenticate":["Bearer realm="security"","ApiKey","Basic realm="security" charset="UTF-8""]}}],"type":"security_exception","reason":"action [cluster:monitor/main] requires authentication","header":{"WWW-Authenticate":["Bearer realm="security"","ApiKey","Basic realm="security" charset="UTF-8""]}},"status":401}

[grumo35]

force monitor privileges into the textbox

[newKibanaUser]

I did that as well and tried giving "all" privileges. nothing works.

If I use plain username and password, it works. cloud.id and cloud.auth does not work.

[grumo35]

401 look like you maybe miss typed password or check for wrong endpoint url.

[newKibanaUser]

no still same. just using the some simple password and I double checked the Cloud ID (took the cloud id under my deployment name).

[newKibanaUser]

I used the same username and password as below and it works perfectly. can't figure out why throwing 401 while using in cloud.auth

output.elasticsearch:
hosts: ["https://xxxxxxxx:9999"]
username: "newuser"
password: "newuser"

[Alex_Piggott]

Hi @newKibanaUser

If you're using cloud.id/cloud.auth I think you have to remove all of hosts/username/password fields from output.elasticsearch - it wasn't clear from your snippets if you were doing that

If that doesn't help, can you run the second half of the cloud id through a base64 decoder and confirm that it is the URL (including port) you are expecting?

Alex

[newKibanaUser]

Hi Alex-

If I remove hosts, it gives below error so I have to include the hosts.

ERROR instance/beat.go:916 Exiting: error initializing publisher: missing required field accessing 'output.elasticsearch.hosts'

I used online tool to decode cloud.id (without the deployment name), but the results are flipped. Is that normal?

for cloud ID "aaaaaaaaaaaa", result is like below. The URL after second dollar sign does not match with my original URL.

  non-production.ece.iaas-test:9243$aaaaaaaaa$another set of string

Here is my YML file.

[Alex_Piggott]

Oh cloud.id and cloud.auth live at the top level I think, not under output.elasticsearch?

[newKibanaUser]

you nailed it. It works now. Thanks.

Another follow-up question before we can close this. The filebeat gives below error and I am OK for now since I am not using modules in filebeat yet.

ERROR fileset/modules.go:125 Not loading modules. Module directory not found: /usr/share/filebeat/bin/module

Actually the modules resides in the /usr/share/filebeat/module directory at my instance (not under bin). How to tell filebeat to look into this particular folder for modules (or) move those modules to the folder it is looking?

[Alex_Piggott]

I suggest asking that question in a different thread so that people who know lots about beats but not about beats/cloud aren't put off by the title (and vice versa!)

It seems like there are already some similar Q/As kicking around for that though - eg "Module directory not found" - filebeat looking in wrong place?