Filebeat googlecloud module not loading service account

Elastic Stack Beats
,
1

I am running into an issue where filebeat is unable to load my gcp service account keyfile.

This is my googlecloud config file -

 - module: googlecloud
      audit:
        enabled: true
        var.project_id: my-project
        var.topic: filebeat-audited-resource-topic
        var.subscription_name: filebeat-audited-resource-subscription 
        var.credentials_file: ${path.config}/credentials.json

Originally I thought the issue was with the key file not existing or the whitespace but I'm able to use the keyfile with gcloud. -

root@ip-10-128-1-69:/home/ubuntu# ls -al /usr/share/filebeat/credentials.json 
-rw-r--r-- 1 root root 2315 Sep 15 19:51 /usr/share/filebeat/credentials.json
root@ip-10-128-1-69:/home/ubuntu# gcloud auth activate-service-account --key-file /usr/share/filebeat/credentials.json
Activated service account credentials for: [elastic-beats@another-project.iam.gserviceaccount.com]
root@ip-10-128-1-69:/home/ubuntu# gcloud config set project my-project
Updated property [core/project].
WARNING: You do not appear to have access to project [my-project] or it does not exist.
root@ip-10-128-1-69:/home/ubuntu# gcloud pubsub topics list-subscriptions  filebeat-audited-resource-topic
---
  projects/my-project/subscriptions/filebeat-audited-resource-subscription

Log file -

2020-09-15T20:20:43.330Z        ERROR   fileset/factory.go:103  Error creating input: no authentication credentials were configured or detected (credentials_file, credentials_json, and application default credentials (ADC)) accessing config
2020-09-15T20:20:43.330Z        INFO    beater/crawler.go:148   Stopping Crawler
2020-09-15T20:20:43.330Z        INFO    beater/crawler.go:158   Stopping 0 inputs
2020-09-15T20:20:43.330Z        INFO    beater/crawler.go:178   Crawler stopped
2020-09-15T20:20:43.331Z        INFO    [registrar]     registrar/registrar.go:131      Stopping Registrar
2020-09-15T20:20:43.331Z        INFO    [registrar]     registrar/registrar.go:165      Ending Registrar
2020-09-15T20:20:43.332Z        DEBUG   [registrar]     registrar/registrar.go:166      Stopping Registrar
2020-09-15T20:20:43.332Z        INFO    [registrar]     registrar/registrar.go:136      Registrar stopped
2020-09-15T20:20:43.334Z        INFO    [monitoring]    log/log.go:154  Uptime: 91.292571ms
2020-09-15T20:20:43.334Z        INFO    [monitoring]    log/log.go:131  Stopping metrics logging.
2020-09-15T20:20:43.335Z        INFO    instance/beat.go:456    filebeat stopped.
2020-09-15T20:20:43.335Z        ERROR   instance/beat.go:951    Exiting: Failed to start crawler: creating module reloader failed: no authentication credentials were configured or detected (credentials_file, credentials_json, and application default credentials (ADC)) accessing config
Exiting: Failed to start crawler: creating module reloader failed: no authentication credentials were configured or detected (credentials_file, credentials_json, and application default credentials (ADC)) accessing config

Am I using the wrong type of key or do I need to make changes to it for filebeats to be able to use it correctly?

2

Could you please post your config file? It looks like misconfiguration issue.

3

This is my filebeat.yml

cloud.auth: my-cloud-auth
cloud.id: my-cloud
filebeat:
  config.modules:
    enabled: true
    path: ${path.config}/modules.d/*.yml
setup.kibana:
  host: https://my-kibana-host

output:
  elasticsearch:
    hosts:
    - https://my-host
    password: my-password
    username: my-username

logging:
  files:
    rotateeverybytes: 10485760
4

To check if the service account was working correctly I configured metricbeat's googlecloud module with the same keyfile and it works correctly.

5

Any new information on this? Thanks

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.