Functionbeat missing package io/fs

Elastic Stack Beats
1

Hello,

Since 7.15, deployment for functionbeat on GCP cloud function leads to an error :

Function: functionbeat-XXXXX, could not update, error: error while creating function (code: 3):

[89](/-/jobs/1615821324#L89)Build failed: src/pubsub/vendor/github.com/elastic/beats/v7/libbeat/metric/system/cgroup/cgv2/io_helper_linux.go:23:2: cannot find package "io/fs" in any of:

[90](/-/jobs/1615821324#L90) /workspace/src/pubsub/vendor/io/fs (vendor tree)

[91](/-/jobs/1615821324#L91) /usr/local/go/src/io/fs (from $GOROOT)

[92](/-/jobs/1615821324#L92) /workspace/src/io/fs (from $GOPATH); Error ID: 6191efcd

Do you guys have any idea how to resolve this issue ?

Thanks !

2

Hi @Yero_Me , welcome to the Elastic Stack Community forum.

May you please provide more information to help troubleshoot this issue?

Which version of go are you using to build the code?
Are you using go modules?

3

Hello @Edoardo_Tenani ,

The code is built within my CI/CD integration. I'm using the docker image "google/cloud-sdk:latest". During the deployment process I'm downloading the lastest version of functionbeat 7.15.0-linux-x86_64. Then I "tar xzf" the downloaded file.

The functionbeat is deployed using the "deploy" or "update" command (Functionbeat command reference | Functionbeat Reference [7.15] | Elastic)

Full command is here :

./functionbeat -v -e -E cloud.id=$ES_CLOUD_ID -E cloud.auth=$ES_RECORD_CLOUD_AUTH -E monitoring.cloud.id=$ES_CLOUD_ID -E monitoring.cloud.auth=$ES_RECORD_CLOUD_AUTH -d "*" update functionbeat-record-provisioner

The deployment process works fine with functionbeat 1.11.2. Starting with functionbeat 1.12, I was facing this issue reported here : https://github.com/elastic/beats/issues/24925 up until version 1.14.

As a workaround, I tried to deploy functionbeat 1.11.2 (which automaticaly run on GCP cloud function with runtime -> Go 1.13). Then I've forced the cloud function to Run with Go 1.16. And finaly I've tryied to update functionbeat from 1.11.2 to 1.15.0 but it ends up with the same error.

Do you think I need to install any requirement on the "google/cloud-sdk:latest" for it to work ?

PS : my yml configuration file is working fine with functionbeat 1.11.

Thank you for your help.

4

Thanks for the detailed answer!

I'll check internally if this is a known issue. It seems related to the Go version in use as the io/fs module was added in go 1.16.

If that was the issue you should be able to workaround it by running functionbeat 7.15 with go 1.16. May you try it out?

Thanks!

5

Thanks for the reply.

I don't think I have any option to run functionbeat 1.15 with go 1.16. This version of Go is just released by GCP and it's still in preview (whatever that mean).

There is a good chance functionbeat deploy command will install functionbeat with the default runtime which is Go 1.13. Like I say, I didn't find any configuration parameter to force the use of Go 1.16 in functionbeat.yml.

Thanks again

6

Sorry for the double answer.
I've just notice, the setup -e command gives me this :

2021-09-23T19:43:08.174Z	INFO	[beat]	instance/beat.go:1023	Build info	{"system_info": {"build": {"commit": "9023152025ec6251bc6b6c38009b309157f10f17", "libbeat": "7.15.0", "time": "2021-09-16T02:32:20.000Z", "version": "7.15.0"}}}
2021-09-23T19:43:08.175Z	INFO	[beat]	instance/beat.go:1026	Go runtime info	{"system_info": {"go": {"os":"linux","arch":"amd64","max_procs":1,"version":"go1.16.6"}}}
2021-09-23T19:43:08.175Z	INFO	[beat]	instance/beat.go:1030	Host info	{"system_info": {"host": {"architecture":"x86_64","boot_time":"2021-09-23T19:02:01Z","containerized":true,"name":"runner-z3wu8uu--project-25888153-concurrent-0","ip":["REDACTED"],"kernel_version":"5.4.109+","mac":["REDACTED"],"os":{"type":"linux","family":"debian","platform":"debian","name":"Debian GNU/Linux","version":"10 (buster)","major":10,"minor":0,"patch":0,"codename":"buster"},"timezone":"UTC","timezone_offset_sec":0,"id":"ea1fc30b6f4a173cea015d229c6b55b6"}}}

If I'm correct, I think what might be the issue is functionbeat is building the code with Go 1.16.6 but is deployed on GCP cloud function that run with Go 1.13. Do you know if there is any way whether to build with Go 1.13 or to force the cloud function to work with Go 1.16 (just released) ?

Jérôme

7

Hello @Edoardo_Tenani, did you manage to get some information internally ?
I've tried some workaround, like deploying manually but it didn't work.

Regards,

Jérôme

8

Hello,

yes and I may have a workaround. Is possible to upgrade Go runtime on GCP by changing the runtime constant in beats/template_builder.go at 7e4affb6702a39f378170b8866f966d1b29e06fc · elastic/beats · GitHub

The value to use Golang 1.16 is go116. You can then build functionbeat with mage build (from within the functionbeat/ folder).

This new functionbeat binary should run functions with the updated go runtime, where io/fs is present.

Meanwhile I'll open a GitHub issue to discuss how to address this.

1 Like
9

Hello @Edoardo_Tenani,

Thanks a lot for your advice, it worked for the deployment part. functionbeat is properly deployed on GCP cloud function with Go 1.16.

Unfortunately, after sending some traffic through the cloud function I still get the panic crash error. Same behaviour as before, the first message is properly delivered to my Elastic cloud cluster. I can see it in kibana. The second message always makes functionbeat to crash

FATAL [functionbeat] instance/beat.go:171 Failed due to panic. {"panic": "name version already used", "stack": "pubsub/vendor/github.com/elastic/beats/v7/libbeat/cmd/instance.Run.func1.1\n\t/workspace/src/pubsub/vendor/github.com/elastic/beats/v7/libbeat/cmd/instance/beat.go:172\nruntime.gopanic\n\t/usr/local/go/src/runtime/panic.go:965\npubsub/vendor/github.com/elastic/beats/v7/libbeat/monitoring.panicErr\n\t/workspace/src/pubsub/vendor/github.com/elastic/beats/v7/libbeat/monitoring/registry.go:257\npubsub/vendor/github.com/elastic/beats/v7/libbeat/monitoring.(*Registry).doAdd\n\t/workspace/src/pubsub/vendor/github.com/elastic/beats/v7/libbeat/monitoring/registry.go:159\npubsub/vendor/github.com/elastic/beats/v7/libbeat/monitoring.addVar\n\t/workspace/src/pubsub/vendor/github.com/elastic/beats/v7/libbeat/monitoring/metrics.go:225\npubsub/vendor/github.com/elastic/beats/v7/libbeat/monitoring.NewString\n\t/workspace/src/pubsub/vendor/github.com/elastic/beats/v7/libbeat/monitoring/metrics.go:170\npubsub/vendor/github.com/elastic/beats/v7/libbeat/cmd/instance.Run.func1\n\t/workspace/src/pubsub/vendor/github.com/elastic/beats/v7/libbeat/cmd/instance/beat.go:182\npubsub/vendor/github.com/elastic/beats/v7/libbeat/cmd/instance.Run\n\t/workspace/src/pubsub/vendor/github.com/elastic/beats/v7/libbeat/cmd/instance/beat.go:197\npubsub.RunPubSub\n\t/workspace/src/pubsub/pubsub.go:35\nreflect.Value.call\n\t/usr/local/go/src/reflect/value.go:476\nreflect.Value.Call\n\t/usr/local/go/src/reflect/value.go:337\ngithub.com/GoogleCloudPlatform/functions-framework-go/funcframework.runUserFunctionWithContext\n\t/workspace/src/github.com/GoogleCloudPlatform/functions-framework-go/funcframework/framework.go:202\ngithub.com/GoogleCloudPlatform/functions-framework-go/funcframework.runBackgroundEvent\n\t/workspace/src/github.com/GoogleCloudPlatform/functions-framework-go/funcframework/events.go:101\ngithub.com/GoogleCloudPlatform/functions-framework-go/funcframework.handleEventFunction\n\t/workspace/src/github.com/GoogleCloudPlatform/functions-framework-go/funcframework/framework.go:169\ngithub.com/GoogleCloudPlatform/functions-framework-go/funcframework.registerEventFunction.func1\n\t/workspace/src/github.com/GoogleCloudPlatform/functions-framework-go/funcframework/framework.go:136\nnet/http.HandlerFunc.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2049\nnet/http.(*ServeMux).ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2428\nnet/http.serverHandler.ServeHTTP\n\t/usr/local/go/src/net/http/server.go:2867\nnet/http.(*conn).serve\n\t/usr/local/go/src/net/http/server.go:1932"}

I'm facing this issue since 7.11 unfortunatly, I cannot manage to make functionbeat to work properly. Issue has been raised about it : GCP/FunctionBeat Errors [cannot find matching process for pid=1, name version already used] · Issue #25596 · elastic/beats · GitHub but it didn't get much love for the moment :frowning:

Maybe you can try to revive it otherwise thanks a lot for your help.

Jérôme

10

I have good news! There is an open pr (thanks @kvch!) to proper update functionbeat to go 1.16.

There is another solution we announced recently that could bypass functionbeat altogether, that is using GCP Dataflow to forward logs to Elasticsearch. You can read more in this announcement blog post. With it you can route logs to GCP Pub/Sub and then forward them to your public routable Elasticsearch cluster with a GCP Dataflow Job.
I worked on a Terraform based example for GCP Audit logs, you may want to check that out too.

11

Hello Edoardo,

The dataflow option sounds good. I'll give it a try tonight. What are the pros of using Functionbeat versus Dataflow now that the template is released ?

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

13

@Yero_Me I'm sorry I missed your response notification.

What are the pros of using Functionbeat versus Dataflow now that the template is released ?

I don't think there is any pros, if not familiarity. Overall the Dataflow templates is more robust and scalable and overcome the shortcomings of functionbeat. The main disadvantage may be cost, but it really depends on usage patterns.

If you tried it I'd be happy to know your opinion on it (PMs are open if you prefer that way).