Functionbeat GCP pub/sub requiring storage entry point when only pub/sub is enabled

[Sebastian_Borys]

Deploying functionbeat on GCP with entry point RunPubSub returns an error complaining about storage entry point, which I am not using.

I get the following error upon deploy

Deployment failure:
Build failed: # functions.local/app/main
src/functions.local/app/main/main.go:43:21: undefined: storage.RunPubSub; Error ID: 2f5e35a0

The problem is ,as I noticed it, Functionbeat is requiring storage entry point. I am NOT using the gcp storage functionality, I am using pub/sub functionality with RunPubSub. Why is it requiring the storage entry point? I am not able to provide multiple entry points to my knowledge.

• Version: 7.13.4 ( noticed on others as well)
• Operating System: LINUX X86_64
• Github Issue URL: Functionbeat GCP pub/sub requiring storage entry point when only pub/sub is enabled · Issue #29967 · elastic/beats · GitHub
• Steps to Reproduce:

1. Package functionbeat using LINUX X86_64 7.13.4 up using the following command

./functionbeat  export function pubsub -c functionbeat-file.yml

2. Go into GCP function and go through creating a cloud function steps, upload the packaed zip,
3. add entry point "RunPubSub"
4. Select Golang version 1.16 in order to avoid another bug (functionbeat 7.12 can't deploy -- unknown field 'VerifyConnection' in struct literal of type tls.Config · Issue #24925 · elastic/beats · GitHub). <<< Selecting 1.13 will yield this error. Keep in mind the runtime of the function is GoLang 1.13 (could this be the problem?)

 ./functionbeat export function pubsub |grep runtime
    runtime: go113

My config file

functionbeat.provider.gcp.location_id: "us-central1"

functionbeat.provider.gcp.project_id: "some-project"

functionbeat.provider.gcp.storage_name: "some-bucket"

functionbeat.provider.gcp.functions:
  - name: pubsub
    enabled: true
    type: pubsub

    description: "Google Cloud Function for Pub/Sub"
    memory_size: 512MB
    timeout: 60s
    service_account_email: some-email

    trigger:
      resource: "some-topic"
processors:
- add_cloud_metadata:
    providers: ["gcp"]
- add_fields:
    target: field
    fields:
      sourcetype: test
      service: logging
- drop_event:
    when:
      not:
       regexp:
         field.sourcetype: ".*"
logging.level: info
logging.json: true
output.logstash:
  hosts: ["some-host"]
  pipelining: 0
  max_retries: 0
  ttl: 30s
setup.ilm.enabled: false
setup.template.enabled: false
path:
  config: ./
  home: ./

Relating: Functionbeat missing package io/fs - #10 by Edoardo_Tenani

[Sebastian_Borys]

I wonder even if I manage to get this to run if I will run into this

Functionbeat missing package io/fs

Hello @Edoardo_Tenani, Thanks a lot for your advice, it worked for the deployment part. functionbeat is properly deployed on GCP cloud function with Go 1.16. Unfortunately, after sending some traffic through the cloud function I still get the panic crash error. Same behaviour as before, the first message is properly delivered to my Elastic cloud cluster. I can see it in kibana. The second message always makes functionbeat to crash FATAL [functionbeat] instance/beat.go:171 Failed due to panic.…

GCP/FunctionBeat Errors [cannot find matching process for pid=1, name version already used] · Issue #25596 · elastic/beats · GitHub 1

[Sebastian_Borys]

Keep in mind the runtime of the function is GoLang 1.13 (could this be the problem?)
recompiled the functionbeat binary as goLang 1.16 and repackaged, same error.

[Yero_Me]

You should avoid getting functionbeat to run on cloud function. It doesn't work and it will never reach a proper release. Edoardo told me about using a GCP dataflow template and it's working fine so far. The downside is the cost tho.

[Sebastian_Borys]

Thanks, - just surprised that they included gcp support for so long until 7.15