# FILEBEAT - GOOGLE CLOUD MODULE - PUB/SUB

**URL:** <https://discuss.elastic.co/t/filebeat-google-cloud-module-pub-sub/244804>\
**Category:** Beats\
**Tags:** beats-module, filebeat\
**Created:** [August 13, 2020, 6:24am UTC](https://discuss.elastic.co/t/filebeat-google-cloud-module-pub-sub/244804 "2020-08-13T06:24:07Z")\
**Posts on this page:** 8\
**Page:** 1

<div class="post-metadata">

**Author:** ![juaromu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/juaromu/32/73805_2.png) [@juaromu](https://discuss.elastic.co/u/juaromu)\
**Post date:** [August 13, 2020, 6:24am UTC](https://discuss.elastic.co/t/filebeat-google-cloud-module-pub-sub/244804/1 "2020-08-13T06:24:07Z")

</div>

Hi there:

Running Filebeat, 7.8.1 and using the Google Cloud Module, PUB/SUB, to pull out VPC Firewall Logs.

Getting an error (I believe JSON related) when Filebeat starts:

`ERROR [google.pubsub] googlepubsub/input.go:127 pubsub: invalid character '\n' in string literal`

The Filbeat config file looks like this:

```auto
    filebeat.inputs:
    - type: google-pubsub
      project_id: redacted
      topic: vpc-firewall-logs-topic
      subscription.name: filebeat-vpc-firewall-logs-sub
      credentials_file: ${path.config}/redacted.json
      subscription.create: false
      enabled: true
    filebeat.config.modules:
      path: ${path.config}/modules.d/*.yml
      reload.enabled: false
      document_type: json
      json.keys_under_root: true
      json.add_error_key: true
    output.logstash:
      hosts: ["redacted:5045"]
      ssl.enabled: true
      client_authentication: none

```

Any ideas?

Thanks!

Juan.

---

<div class="post-metadata">

**Author:** ![kvch](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kvch/32/72058_2.png) [@kvch](https://discuss.elastic.co/u/kvch)\
**Post date:** [August 13, 2020, 11:59am UTC](https://discuss.elastic.co/t/filebeat-google-cloud-module-pub-sub/244804/2 "2020-08-13T11:59:36Z")

</div>

Could you please share debug logs of Filebeat (output of `./filebeat -e -d "*"`)?

---

<div class="post-metadata">

**Author:** ![juaromu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/juaromu/32/73805_2.png) [@juaromu](https://discuss.elastic.co/u/juaromu)\
**Post date:** [August 13, 2020, 9:33pm UTC](https://discuss.elastic.co/t/filebeat-google-cloud-module-pub-sub/244804/3 "2020-08-13T21:33:03Z")

</div>

Hi Noemi:

Thanks for your response.

Output attached here (several entries due to characters limit here):

```auto
2020-08-14T07:24:08.830+1000 INFO instance/beat.go:647 Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/var/lib/filebeat] Logs path: [/var/log/filebeat]
2020-08-14T07:24:08.830+1000 DEBUG [beat] instance/beat.go:699 Beat metadata path: /var/lib/filebeat/meta.json
2020-08-14T07:24:08.830+1000 INFO instance/beat.go:655 Beat ID: 34bdb566-4c8d-43fd-8938-9efbb12f69c7
2020-08-14T07:24:08.831+1000 DEBUG [docker] docker/client.go:48 Docker client will negotiate the API version on the first request.
2020-08-14T07:24:08.833+1000 DEBUG [add_docker_metadata] add_docker_metadata/add_docker_metadata.go:87 add_docker_metadata: docker environment not detected: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
2020-08-14T07:24:08.833+1000 DEBUG [add_cloud_metadata] add_cloud_metadata/providers.go:126 add_cloud_metadata: starting to fetch metadata, timeout=3s
2020-08-14T07:24:08.835+1000 DEBUG [kubernetes] add_kubernetes_metadata/kubernetes.go:138 Could not create kubernetes client using in_cluster config: unable to build kube config due to error: invalid configuration: no configuration has been provided {"libbeat.processor": "add_kubernetes_metadata"}
2020-08-14T07:24:08.839+1000 DEBUG [add_cloud_metadata] add_cloud_metadata/providers.go:162 add_cloud_metadata: received disposition for digitalocean after 5.930874ms. result=[provider:digitalocean, error=failed with http status code 404, metadata={}]
2020-08-14T07:24:08.840+1000 DEBUG [add_cloud_metadata] add_cloud_metadata/providers.go:162 add_cloud_metadata: received disposition for openstack after 6.284636ms. result=[provider:openstack, error=failed with http status code 404, metadata={}]
2020-08-14T07:24:08.840+1000 DEBUG [add_cloud_metadata] add_cloud_metadata/providers.go:162 add_cloud_metadata: received disposition for az after 6.582425ms. result=[provider:az, error=failed with http status code 404, metadata={}]
2020-08-14T07:24:08.840+1000 DEBUG [add_cloud_metadata] add_cloud_metadata/providers.go:162 add_cloud_metadata: received disposition for aws after 6.870747ms. result=[provider:aws, error=failed with http status code 404, metadata={}]
2020-08-14T07:24:08.842+1000 DEBUG [add_cloud_metadata] add_cloud_metadata/providers.go:162 add_cloud_metadata: received disposition for gcp after 8.349071ms. result=[provider:gcp, error=<nil>, metadata={"availability_zone":"australia-southeast1-b","instance":{"id":"6552887070891700583","name":"jumphost"},"machine":{"type":"n1-standard-1"},"project":{"id":"redacted"},"provider":"gcp"}]
2020-08-14T07:24:08.842+1000 DEBUG [add_cloud_metadata] add_cloud_metadata/providers.go:129 add_cloud_metadata: fetchMetadata ran for 8.398806ms
2020-08-14T07:24:08.842+1000 INFO [add_cloud_metadata] add_cloud_metadata/add_cloud_metadata.go:93 add_cloud_metadata: hosting provider type detected as gcp, metadata={"availability_zone":"australia-southeast1-b","instance":{"id":"6552887070891700583","name":"jumphost"},"machine":{"type":"n1-standard-1"},"project":{"id":"redacted"},"provider":"gcp"}
2020-08-14T07:24:08.842+1000 DEBUG [processors] processors/processor.go:101 Generated new processors: add_host_metadata=[netinfo.enabled=[true], cache.ttl=[5m0s]], add_cloud_metadata={"availability_zone":"australia-southeast1-b","instance":{"id":"6552887070891700583","name":"jumphost"},"machine":{"type":"n1-standard-1"},"project":{"id":"redacted"},"provider":"gcp"}, add_docker_metadata=[match_fields=[] match_pids=[process.pid, process.ppid]], add_kubernetes_metadata
2020-08-14T07:24:08.842+1000 DEBUG [seccomp] seccomp/seccomp.go:117 Loading syscall filter {"seccomp_filter": {"no_new_privs":true,"flag":"tsync","policy":{"default_action":"errno","syscalls":[{"names":["accept","accept4","access","arch_prctl","bind","brk","chmod","chown","clock_gettime","clone","close","connect","dup","dup2","epoll_create","epoll_create1","epoll_ctl","epoll_pwait","epoll_wait","exit","exit_group","fchdir","fchmod","fchmodat","fchown","fchownat","fcntl","fdatasync","flock","fstat","fstatfs","fsync","ftruncate","futex","getcwd","getdents","getdents64","geteuid","getgid","getpeername","getpid","getppid","getrandom","getrlimit","getrusage","getsockname","getsockopt","gettid","gettimeofday","getuid","inotify_add_watch","inotify_init1","inotify_rm_watch","ioctl","kill","listen","lseek","lstat","madvise","mincore","mkdirat","mmap","mprotect","munmap","nanosleep","newfstatat","open","openat","pipe","pipe2","poll","ppoll","pread64","pselect6","pwrite64","read","readlink","readlinkat","recvfrom","recvmmsg","recvmsg","rename","renameat","rt_sigaction","rt_sigprocmask","rt_sigreturn","sched_getaffinity","sched_yield","sendfile","sendmmsg","sendmsg","sendto","set_robust_list","setitimer","setsockopt","shutdown","sigaltstack","socket","splice","stat","statfs","sysinfo","tgkill","time","tkill","uname","unlink","unlinkat","wait4","waitid","write","writev"],"action":"allow"}]}}}
2020-08-14T07:24:08.842+1000 INFO [seccomp] seccomp/seccomp.go:124 Syscall filter successfully installed
2020-08-14T07:24:08.842+1000 INFO [beat] instance/beat.go:983 Beat info {"system_info": {"beat": {"path": {"config": "/etc/filebeat", "data": "/var/lib/filebeat", "home": "/usr/share/filebeat", "logs": "/var/log/filebeat"}, "type": "filebeat", "uuid": "34bdb566-4c8d-43fd-8938-9efbb12f69c7"}}}
2020-08-14T07:24:08.842+1000 INFO [beat] instance/beat.go:992 Build info {"system_info": {"build": {"commit": "94f7632be5d56a7928595da79f4b829ffe123744", "libbeat": "7.8.1", "time": "2020-07-21T15:12:45.000Z", "version": "7.8.1"}}}
2020-08-14T07:24:08.842+1000 INFO [beat] instance/beat.go:995 Go runtime info {"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":1,"version":"go1.13.10"}}}
2020-08-14T07:24:08.843+1000 INFO [beat] instance/beat.go:999 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-07-18T11:25:35+10:00","containerized":false,"name":"jumphost","ip":["127.0.0.1/8","::1/128","10.255.0.3/32","fe80::4001:aff:feff:3/64"],"kernel_version":"4.9.0-12-amd64","mac":["42:01:0a:ff:00:03"],"os":{"family":"debian","platform":"debian","name":"Debian GNU/Linux","version":"9 (stretch)","major":9,"minor":0,"patch":0,"codename":"stretch"},"timezone":"AEST","timezone_offset_sec":36000,"id":"68d8fd9b514d8e139dca1168ba4b3300"}}}

```

---

<div class="post-metadata">

**Author:** ![juaromu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/juaromu/32/73805_2.png) [@juaromu](https://discuss.elastic.co/u/juaromu)\
**Post date:** [August 13, 2020, 9:33pm UTC](https://discuss.elastic.co/t/filebeat-google-cloud-module-pub-sub/244804/4 "2020-08-13T21:33:37Z")

</div>

```auto
2020-08-14T07:24:08.843+1000 INFO [beat] instance/beat.go:1028 Process info {"system_info": {"process": {"capabilities": {"inheritable":null,"permitted":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"effective":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"bounding":["chown","dac_override","dac_read_search","fowner","fsetid","kill","setgid","setuid","setpcap","linux_immutable","net_bind_service","net_broadcast","net_admin","net_raw","ipc_lock","ipc_owner","sys_module","sys_rawio","sys_chroot","sys_ptrace","sys_pacct","sys_admin","sys_boot","sys_nice","sys_resource","sys_time","sys_tty_config","mknod","lease","audit_write","audit_control","setfcap","mac_override","mac_admin","syslog","wake_alarm","block_suspend","audit_read"],"ambient":null}, "cwd": "/home/jromero", "exe": "/usr/share/filebeat/bin/filebeat", "name": "filebeat", "pid": 15072, "ppid": 15042, "seccomp": {"mode":"filter"}, "start_time": "2020-08-14T07:24:07.870+1000"}}}
2020-08-14T07:24:08.843+1000 INFO instance/beat.go:310 Setup Beat: filebeat; Version: 7.8.1
2020-08-14T07:24:08.843+1000 DEBUG [beat] instance/beat.go:336 Initializing output plugins
2020-08-14T07:24:08.845+1000 DEBUG [publisher] pipeline/consumer.go:137 start pipeline event consumer
2020-08-14T07:24:08.845+1000 INFO [publisher] pipeline/module.go:113 Beat name: jumphost
2020-08-14T07:24:08.847+1000 WARN beater/filebeat.go:156 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2020-08-14T07:24:08.847+1000 INFO instance/beat.go:463 filebeat start running.
2020-08-14T07:24:08.847+1000 DEBUG [test] registrar/migrate.go:159 isFile(/var/lib/filebeat/registry) -> false
2020-08-14T07:24:08.847+1000 DEBUG [test] registrar/migrate.go:159 isFile() -> false
2020-08-14T07:24:08.848+1000 DEBUG [test] registrar/migrate.go:152 isDir(/var/lib/filebeat/registry/filebeat) -> true
2020-08-14T07:24:08.848+1000 DEBUG [test] registrar/migrate.go:159 isFile(/var/lib/filebeat/registry/filebeat/meta.json) -> true
2020-08-14T07:24:08.848+1000 DEBUG [registrar] registrar/migrate.go:51 Registry type '0' found
2020-08-14T07:24:08.848+1000 DEBUG [registrar] registrar/registrar.go:125 Registry file set to: /var/lib/filebeat/registry/filebeat/data.json
2020-08-14T07:24:08.848+1000 WARN beater/filebeat.go:339 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2020-08-14T07:24:08.848+1000 INFO registrar/registrar.go:145 Loading registrar data from /var/lib/filebeat/registry/filebeat/data.json
2020-08-14T07:24:08.848+1000 INFO registrar/registrar.go:152 States Loaded from registrar: 0
2020-08-14T07:24:08.848+1000 INFO [crawler] beater/crawler.go:71 Loading Inputs: 1
2020-08-14T07:24:08.848+1000 INFO [google.pubsub] googlepubsub/input.go:112 Initialized Google Pub/Sub input. {"pubsub_project": "redacted", "pubsub_topic": "vpc-firewall-logs-topic", "pubsub_subscription": {"Name":"filebeat-vpc-firewall-logs-sub","NumGoroutines":1,"MaxOutstandingMessages":1000,"Create":false}}
2020-08-14T07:24:08.848+1000 INFO [crawler] beater/crawler.go:141 Starting input (ID: 11448794657429154599)
2020-08-14T07:24:08.848+1000 DEBUG [cfgfile] cfgfile/reload.go:132 Checking module configs from: /etc/filebeat/modules.d/*.yml
2020-08-14T07:24:08.849+1000 DEBUG [cfgfile] cfgfile/cfgfile.go:193 Load config from file: /etc/filebeat/modules.d/googlecloud.yml
2020-08-14T07:24:08.849+1000 DEBUG [cfgfile] cfgfile/reload.go:146 Number of module configs found: 1
2020-08-14T07:24:08.857+1000 DEBUG [processor.javascript] javascript/session.go:162 Registered params with processor
2020-08-14T07:24:08.858+1000 DEBUG [processors] processors/processor.go:101 Generated new processors: script=[type=javascript, id=, sources=/usr/share/filebeat/module/googlecloud/firewall/config/pipeline.js], add_fields={"ecs":{"version":"1.5.0"}}
2020-08-14T07:24:08.858+1000 INFO [google.pubsub] googlepubsub/input.go:112 Initialized Google Pub/Sub input. {"pubsub_project": "redacted", "pubsub_topic": "projects/redacted/topics/vpc-firewall-logs-topic", "pubsub_subscription": {"Name":"projects/redacted/subscriptions/filebeat-vpc-firewall-logs-sub","NumGoroutines":1,"MaxOutstandingMessages":1000,"Create":true}}
2020-08-14T07:24:08.858+1000 INFO [crawler] beater/crawler.go:108 Loading and starting Inputs completed. Enabled inputs: 1
2020-08-14T07:24:08.859+1000 INFO [monitoring] log/log.go:118 Starting metrics logging every 30s
2020-08-14T07:24:08.859+1000 DEBUG [registrar] registrar/registrar.go:278 Starting Registrar
2020-08-14T07:24:08.859+1000 INFO [google.pubsub] googlepubsub/input.go:122 Pub/Sub input worker has started. {"pubsub_project": "redacted", "pubsub_topic": "vpc-firewall-logs-topic", "pubsub_subscription": {"Name":"filebeat-vpc-firewall-logs-sub","NumGoroutines":1,"MaxOutstandingMessages":1000,"Create":false}}
2020-08-14T07:24:08.859+1000 ERROR [google.pubsub] googlepubsub/input.go:127 pubsub: invalid character '\n' in string literal {"pubsub_project": "redacted", "pubsub_topic": "vpc-firewall-logs-topic", "pubsub_subscription": {"Name":"filebeat-vpc-firewall-logs-sub","NumGoroutines":1,"MaxOutstandingMessages":1000,"Create":false}}
2020-08-14T07:24:08.859+1000 INFO [google.pubsub] googlepubsub/input.go:128 Pub/Sub input worker has stopped. {"pubsub_project": "redacted", "pubsub_topic": "vpc-firewall-logs-topic", "pubsub_subscription": {"Name":"filebeat-vpc-firewall-logs-sub","NumGoroutines":1,"MaxOutstandingMessages":1000,"Create":false}}
2020-08-14T07:24:08.859+1000 INFO cfgfile/reload.go:164 Config reloader started
2020-08-14T07:24:08.860+1000 DEBUG [cfgfile] cfgfile/reload.go:194 Scan for new config files
2020-08-14T07:24:08.860+1000 DEBUG [cfgfile] cfgfile/cfgfile.go:193 Load config from file: /etc/filebeat/modules.d/googlecloud.yml
2020-08-14T07:24:08.861+1000 DEBUG [cfgfile] cfgfile/reload.go:213 Number of module configs found: 1
2020-08-14T07:24:08.862+1000 DEBUG [reload] cfgfile/list.go:62 Starting reload procedure, current runners: 0
2020-08-14T07:24:08.862+1000 DEBUG [reload] cfgfile/list.go:80 Start list: 1, Stop list: 0
2020-08-14T07:24:08.870+1000 DEBUG [processor.javascript] javascript/session.go:162 Registered params with processor

```

---

<div class="post-metadata">

**Author:** ![juaromu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/juaromu/32/73805_2.png) [@juaromu](https://discuss.elastic.co/u/juaromu)\
**Post date:** [August 13, 2020, 9:34pm UTC](https://discuss.elastic.co/t/filebeat-google-cloud-module-pub-sub/244804/5 "2020-08-13T21:34:13Z")

</div>

```auto
2020-08-14T07:24:08.870+1000 DEBUG [processors] processors/processor.go:101 Generated new processors: script=[type=javascript, id=, sources=/usr/share/filebeat/module/googlecloud/firewall/config/pipeline.js], add_fields={"ecs":{"version":"1.5.0"}}
2020-08-14T07:24:08.870+1000 INFO [google.pubsub] googlepubsub/input.go:112 Initialized Google Pub/Sub input. {"pubsub_project": "redacted", "pubsub_topic": "projects/redacted/topics/vpc-firewall-logs-topic", "pubsub_subscription": {"Name":"projects/redacted/subscriptions/filebeat-vpc-firewall-logs-sub","NumGoroutines":1,"MaxOutstandingMessages":1000,"Create":true}}
2020-08-14T07:24:08.871+1000 DEBUG [reload] cfgfile/list.go:101 Starting runner: googlecloud (firewall)
2020-08-14T07:24:08.871+1000 INFO cfgfile/reload.go:224 Loading of config files completed.
2020-08-14T07:24:08.871+1000 INFO [google.pubsub] googlepubsub/input.go:122 Pub/Sub input worker has started. {"pubsub_project": "redacted", "pubsub_topic": "projects/redacted/topics/vpc-firewall-logs-topic", "pubsub_subscription": {"Name":"projects/redacted/subscriptions/filebeat-vpc-firewall-logs-sub","NumGoroutines":1,"MaxOutstandingMessages":1000,"Create":true}}
2020-08-14T07:24:08.871+1000 ERROR [google.pubsub] googlepubsub/input.go:127 pubsub: invalid character '\n' in string literal {"pubsub_project": "redacted", "pubsub_topic": "projects/redacted/topics/vpc-firewall-logs-topic", "pubsub_subscription": {"Name":"projects/redacted/subscriptions/filebeat-vpc-firewall-logs-sub","NumGoroutines":1,"MaxOutstandingMessages":1000,"Create":true}}
2020-08-14T07:24:08.871+1000 INFO [google.pubsub] googlepubsub/input.go:128 Pub/Sub input worker has stopped. {"pubsub_project": "redacted", "pubsub_topic": "projects/redacted/topics/vpc-firewall-logs-topic", "pubsub_subscription": {"Name":"projects/redacted/subscriptions/filebeat-vpc-firewall-logs-sub","NumGoroutines":1,"MaxOutstandingMessages":1000,"Create":true}}
2020-08-14T07:24:18.859+1000 DEBUG [input] input/input.go:141 Run input
2020-08-14T07:24:18.871+1000 DEBUG [input] input/input.go:141 Run input
2020-08-14T07:24:28.859+1000 DEBUG [input] input/input.go:141 Run input
2020-08-14T07:24:28.871+1000 DEBUG [input] input/input.go:141 Run input
2020-08-14T07:24:38.861+1000 INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":20,"time":{"ms":28}},"total":{"ticks":120,"time":{"ms":132},"value":120},"user":{"ticks":100,"time":{"ms":104}}},"handles":{"limit":{"hard":1048576,"soft":1024},"open":9},"info":{"ephemeral_id":"fa62c3cb-3eba-4879-b621-b4d31d65b32e","uptime":{"ms":30088}},"memstats":{"gc_next":13918880,"memory_alloc":7348960,"memory_total":20341384,"rss":56692736},"runtime":{"goroutines":27}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0},"reloads":1,"scans":1},"output":{"type":"logstash"},"pipeline":{"clients":2,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"cpu":{"cores":1},"load":{"1":0,"15":0.21,"5":0.08,"norm":{"1":0,"15":0.21,"5":0.08}}}}}}
2020-08-14T07:24:38.861+1000 DEBUG [input] input/input.go:141 Run input
2020-08-14T07:24:38.871+1000 DEBUG [input] input/input.go:141 Run input
^C2020-08-14T07:24:39.425+1000 DEBUG [service] service/service.go:54 Received sigterm/sigint, stopping
2020-08-14T07:24:39.425+1000 INFO beater/filebeat.go:456 Stopping filebeat
2020-08-14T07:24:39.425+1000 INFO beater/crawler.go:148 Stopping Crawler
2020-08-14T07:24:39.425+1000 INFO beater/crawler.go:158 Stopping 1 inputs
2020-08-14T07:24:39.425+1000 INFO cfgfile/reload.go:227 Dynamic config reloader stopped
2020-08-14T07:24:39.425+1000 INFO [reload] cfgfile/list.go:118 Stopping 1 runners ...
2020-08-14T07:24:39.425+1000 DEBUG [reload] cfgfile/list.go:129 Stopping runner: googlecloud (firewall)
2020-08-14T07:24:39.425+1000 INFO [crawler] beater/crawler.go:163 Stopping input: 11448794657429154599
2020-08-14T07:24:39.425+1000 INFO input/input.go:138 input ticker stopped
2020-08-14T07:24:39.425+1000 DEBUG [reload] cfgfile/list.go:131 Stopped runner: googlecloud (firewall)
2020-08-14T07:24:39.425+1000 INFO input/input.go:138 input ticker stopped
2020-08-14T07:24:39.425+1000 INFO beater/crawler.go:178 Crawler stopped
2020-08-14T07:24:39.425+1000 INFO registrar/registrar.go:367 Stopping Registrar
2020-08-14T07:24:39.425+1000 INFO registrar/registrar.go:293 Ending Registrar
2020-08-14T07:24:39.425+1000 DEBUG [registrar] registrar/registrar.go:411 Write registry file: /var/lib/filebeat/registry/filebeat/data.json (0)
2020-08-14T07:24:39.426+1000 DEBUG [publisher] pipeline/client.go:162 client: closing acker
2020-08-14T07:24:39.426+1000 DEBUG [publisher] pipeline/client.go:173 client: done closing acker
2020-08-14T07:24:39.426+1000 DEBUG [publisher] pipeline/client.go:176 client: cancelled 0 events
2020-08-14T07:24:39.426+1000 DEBUG [publisher] pipeline/client.go:147 client: wait for acker to finish
2020-08-14T07:24:39.426+1000 DEBUG [publisher] pipeline/client.go:149 client: acker shut down
2020-08-14T07:24:39.426+1000 DEBUG [publisher] pipeline/client.go:162 client: closing acker
2020-08-14T07:24:39.426+1000 DEBUG [publisher] pipeline/client.go:173 client: done closing acker
2020-08-14T07:24:39.426+1000 DEBUG [publisher] pipeline/client.go:176 client: cancelled 0 events
2020-08-14T07:24:39.426+1000 DEBUG [publisher] pipeline/client.go:147 client: wait for acker to finish
2020-08-14T07:24:39.426+1000 DEBUG [publisher] pipeline/client.go:149 client: acker shut down
2020-08-14T07:24:39.430+1000 DEBUG [registrar] registrar/registrar.go:404 Registry file updated. 0 states written.
2020-08-14T07:24:39.432+1000 INFO [monitoring] log/log.go:153 Total non-zero metrics {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":20,"time":{"ms":28}},"total":{"ticks":120,"time":{"ms":136},"value":120},"user":{"ticks":100,"time":{"ms":108}}},"handles":{"limit":{"hard":1048576,"soft":1024},"open":9},"info":{"ephemeral_id":"fa62c3cb-3eba-4879-b621-b4d31d65b32e","uptime":{"ms":30659}},"memstats":{"gc_next":13918880,"memory_alloc":7777224,"memory_total":20769648,"rss":56692736},"runtime":{"goroutines":14}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0},"reloads":1,"scans":1},"output":{"type":"logstash"},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0},"writes":{"success":1,"total":1}},"system":{"cpu":{"cores":1},"load":{"1":0,"15":0.2,"5":0.08,"norm":{"1":0,"15":0.2,"5":0.08}}}}}}
2020-08-14T07:24:39.432+1000 INFO [monitoring] log/log.go:154 Uptime: 30.660762122s
2020-08-14T07:24:39.432+1000 INFO [monitoring] log/log.go:131 Stopping metrics logging.
2020-08-14T07:24:39.432+1000 INFO instance/beat.go:469 filebeat stopped.

```

---

<div class="post-metadata">

**Author:** ![andrewkroh](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/andrewkroh/32/3784_2.png) [@andrewkroh](https://discuss.elastic.co/u/andrewkroh)\
**Post date:** [August 14, 2020, 3:21pm UTC](https://discuss.elastic.co/t/filebeat-google-cloud-module-pub-sub/244804/6 "2020-08-14T15:21:38Z")

</div>

> [@juaromu](#):
>
> `2020-08-14T07:24:08.848+1000 INFO [google.pubsub] googlepubsub/input.go:112 Initialized Google Pub/Sub input. {"pubsub_project": "redacted", "pubsub_topic": "vpc-firewall-logs-topic", "pubsub_subscription": {"Name":"filebeat-vpc-firewall-logs-sub","NumGoroutines":1,"MaxOutstandingMessages":1000,"Create":false}}`

> [@juaromu](#):
>
> `2020-08-14T07:24:08.858+1000 INFO [google.pubsub] googlepubsub/input.go:112 Initialized Google Pub/Sub input. {"pubsub_project": "redacted", "pubsub_topic": "projects/redacted/topics/vpc-firewall-logs-topic", "pubsub_subscription": {"Name":"projects/redacted/subscriptions/filebeat-vpc-firewall-logs-sub","NumGoroutines":1,"MaxOutstandingMessages":1000,"Create":true}}`

Are you intentionally running two instances of the pub/sub input? You have one in your `filebeat.inputs` in filebeat.yml and you also have the googlecloud module running from modules.d/.

I suspect the error stems from the creation of the pub/sub client within the input. While not 100% clear I think it's happening while reading your credentails JSON file. Can you validate the file with `jq` or some other tool to check that the file contains valid JSON (e.g. `jq . redacted.json`).

---

<div class="post-metadata">

**Author:** ![juaromu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/juaromu/32/73805_2.png) [@juaromu](https://discuss.elastic.co/u/juaromu)\
**Post date:** [August 14, 2020, 9:49pm UTC](https://discuss.elastic.co/t/filebeat-google-cloud-module-pub-sub/244804/7 "2020-08-14T21:49:49Z")

</div>

Hi Andrew:  
Thx for your help.  
The issue seemed to be indeed loading from inputs and modules.d  
Removing it from inputs solved this 🙂  
Thanks again  
Juan.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 11, 2020, 11:49pm UTC](https://discuss.elastic.co/t/filebeat-google-cloud-module-pub-sub/244804/8 "2020-09-11T23:49:57Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.