Filebeat googlecloud module not loading service account

I am running into an issue where filebeat is unable to load my gcp service account keyfile.

This is my googlecloud config file -

 - module: googlecloud
      audit:
        enabled: true
        var.project_id: my-project
        var.topic: filebeat-audited-resource-topic
        var.subscription_name: filebeat-audited-resource-subscription 
        var.credentials_file: ${path.config}/credentials.json

Originally I thought the issue was with the key file not existing or the whitespace but I'm able to use the keyfile with gcloud. -

root@ip-10-128-1-69:/home/ubuntu# ls -al /usr/share/filebeat/credentials.json 
-rw-r--r-- 1 root root 2315 Sep 15 19:51 /usr/share/filebeat/credentials.json
root@ip-10-128-1-69:/home/ubuntu# gcloud auth activate-service-account --key-file /usr/share/filebeat/credentials.json
Activated service account credentials for: [elastic-beats@another-project.iam.gserviceaccount.com]
root@ip-10-128-1-69:/home/ubuntu# gcloud config set project my-project
Updated property [core/project].
WARNING: You do not appear to have access to project [my-project] or it does not exist.
root@ip-10-128-1-69:/home/ubuntu# gcloud pubsub topics list-subscriptions  filebeat-audited-resource-topic
---
  projects/my-project/subscriptions/filebeat-audited-resource-subscription

Log file -

2020-09-15T20:20:43.330Z        ERROR   fileset/factory.go:103  Error creating input: no authentication credentials were configured or detected (credentials_file, credentials_json, and application default credentials (ADC)) accessing config
2020-09-15T20:20:43.330Z        INFO    beater/crawler.go:148   Stopping Crawler
2020-09-15T20:20:43.330Z        INFO    beater/crawler.go:158   Stopping 0 inputs
2020-09-15T20:20:43.330Z        INFO    beater/crawler.go:178   Crawler stopped
2020-09-15T20:20:43.331Z        INFO    [registrar]     registrar/registrar.go:131      Stopping Registrar
2020-09-15T20:20:43.331Z        INFO    [registrar]     registrar/registrar.go:165      Ending Registrar
2020-09-15T20:20:43.332Z        DEBUG   [registrar]     registrar/registrar.go:166      Stopping Registrar
2020-09-15T20:20:43.332Z        INFO    [registrar]     registrar/registrar.go:136      Registrar stopped
2020-09-15T20:20:43.334Z        INFO    [monitoring]    log/log.go:154  Uptime: 91.292571ms
2020-09-15T20:20:43.334Z        INFO    [monitoring]    log/log.go:131  Stopping metrics logging.
2020-09-15T20:20:43.335Z        INFO    instance/beat.go:456    filebeat stopped.
2020-09-15T20:20:43.335Z        ERROR   instance/beat.go:951    Exiting: Failed to start crawler: creating module reloader failed: no authentication credentials were configured or detected (credentials_file, credentials_json, and application default credentials (ADC)) accessing config
Exiting: Failed to start crawler: creating module reloader failed: no authentication credentials were configured or detected (credentials_file, credentials_json, and application default credentials (ADC)) accessing config

Am I using the wrong type of key or do I need to make changes to it for filebeats to be able to use it correctly?

Could you please post your config file? It looks like misconfiguration issue.

This is my filebeat.yml

cloud.auth: my-cloud-auth
cloud.id: my-cloud
filebeat:
  config.modules:
    enabled: true
    path: ${path.config}/modules.d/*.yml
setup.kibana:
  host: https://my-kibana-host

output:
  elasticsearch:
    hosts:
    - https://my-host
    password: my-password
    username: my-username

logging:
  files:
    rotateeverybytes: 10485760

To check if the service account was working correctly I configured metricbeat's googlecloud module with the same keyfile and it works correctly.

Any new information on this? Thanks