Hello,
I'm trying to configure the logstash output on Functionbeat 1.14.0, I found this thread saying this is not supported, however it's two years old and the docs don't mention it:
The lambda was setup with .\functionbeat.exe -v -e -d "*" deploy cloudwatch and was triggered a couple times (then stopped). Regardless, it seems to time out and not upload to logstash. The lambda logs have this loop:
START RequestId: baf51572-de24-4b32-9421-4610bf831000 Version: $LATEST
2021-08-25T16:10:30.444Z INFO instance/beat.go:665 Home path: [C:\Program Files\functionbeat-7.14.0-windows-x86_64] Config path: [C:\Program Files\functionbeat-7.14.0-windows-x86_64] Data path: [/tmp] Logs path: [/tmp/logs]
2021-08-25T16:10:30.446Z INFO instance/beat.go:673 Beat ID: 4378232e-e2bc-4b36-b84e-30aae048ae34
2021-08-25T16:10:30.446Z INFO [seccomp] seccomp/seccomp.go:101 Syscall filter could not be installed because the kernel does not support seccomp
2021-08-25T16:10:30.450Z INFO [beat] instance/beat.go:1014 Beat info
{
"system_info": {
"beat": {
"path": {
"config": "C:\\Program Files\\functionbeat-7.14.0-windows-x86_64",
"data": "/tmp",
"home": "C:\\Program Files\\functionbeat-7.14.0-windows-x86_64",
"logs": "/tmp/logs"
},
"type": "functionbeat",
"uuid": "4378232e-e2bc-4b36-b84e-30aae048ae34"
}
}
}
2021-08-25T16:10:30.450Z INFO [beat] instance/beat.go:1023 Build info
{
"system_info": {
"build": {
"commit": "e127fc31fc6c00fdf8649808f9421d8f8c28b5db",
"libbeat": "7.14.0",
"time": "2021-07-29T20:21:22.000Z",
"version": "7.14.0"
}
}
}
2021-08-25T16:10:30.450Z INFO [beat] instance/beat.go:1026 Go runtime info
{
"system_info": {
"go": {
"os": "linux",
"arch": "amd64",
"max_procs": 2,
"version": "go1.16.6"
}
}
}
2021-08-25T16:10:30.450Z INFO [beat] instance/beat.go:1030 Host info
{
"system_info": {
"host": {
"architecture": "x86_64",
"boot_time": "2021-08-25T15:53:45Z",
"containerized": true,
"name": "169.254.130.53",
"ip": [
"127.0.0.1/8",
"169.254.79.1/32",
"169.254.79.130/30",
"169.254.76.1/23"
],
"kernel_version": "4.14.231-180.360.amzn2.x86_64",
"mac": [
"5e:65:db:1c:2f:9c",
"8e:2e:87:0c:88:24",
"8a:81:f8:c4:0b:3e"
],
"os": {
"type": "linux",
"family": "redhat",
"platform": "amzn",
"name": "Amazon Linux AMI",
"version": "2018.03",
"major": 2018,
"minor": 3,
"patch": 0
},
"timezone": "UTC",
"timezone_offset_sec": 0
}
}
}
2021-08-25T16:10:30.451Z INFO [beat] instance/beat.go:1059 Process info
{
"system_info": {
"process": {
"capabilities": {
"inheritable": null,
"permitted": null,
"effective": null,
"bounding": [
"chown",
"dac_override",
"dac_read_search",
"fowner",
"fsetid",
"kill",
"setgid",
"setuid",
"setpcap",
"linux_immutable",
"net_bind_service",
"net_broadcast",
"net_admin",
"net_raw",
"ipc_lock",
"ipc_owner",
"sys_module",
"sys_rawio",
"sys_chroot",
"sys_ptrace",
"sys_pacct",
"sys_admin",
"sys_boot",
"sys_nice",
"sys_resource",
"sys_time",
"sys_tty_config",
"mknod",
"lease",
"audit_write",
"audit_control",
"setfcap",
"mac_override",
"mac_admin",
"syslog",
"wake_alarm",
"block_suspend",
"audit_read"
],
"ambient": null
},
"cwd": "/var/task",
"exe": "/var/task/functionbeat-aws",
"name": "functionbeat-aw",
"pid": 7,
"ppid": 1,
"seccomp": {
"mode": "filter",
"no_new_privs": true
},
"start_time": "2021-08-25T16:10:29.750Z"
}
}
}
2021-08-25T16:10:30.451Z INFO instance/beat.go:309 Setup Beat: functionbeat; Version: 7.14.0
2021-08-25T16:10:30.451Z INFO [publisher] pipeline/module.go:113 Beat name: 169.254.130.53
2021-08-25T16:10:30.451Z INFO [monitoring] log/log.go:118 Starting metrics logging every 30s
2021-08-25T16:10:30.451Z INFO instance/beat.go:473 functionbeat start running.
2021-08-25T16:10:30.451Z INFO [functionbeat] beater/functionbeat.go:93 Functionbeat is running
2021-08-25T16:10:30.451Z INFO [functionbeat] beater/functionbeat.go:99 Functionbeat is configuring enabled functions: cloudwatch
2021-08-25T16:10:30.452Z INFO [coordinator.Coordinator] core/coordinator.go:86 The function 'cloudwatch_logs' is starting
2021-08-25T16:10:31.469Z INFO [add_cloud_metadata] add_cloud_metadata/add_cloud_metadata.go:101 add_cloud_metadata: hosting provider type not detected.
2021-08-25T16:10:31.469Z INFO [publisher_pipeline_output] pipeline/output.go:143 Connecting to backoff(tcp://x.x.x.x:5044)
2021-08-25T16:10:31.470Z INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
2021-08-25T16:10:31.470Z INFO [publisher] pipeline/retry.go:223 done
END RequestId: baf51572-de24-4b32-9421-4610bf831000
REPORT RequestId: baf51572-de24-4b32-9421-4610bf831000 Duration: 3000.53 ms Billed Duration: 3000 ms Memory Size: 128 MB Max Memory Used: 127 MB Init Duration: 600.67 ms
2021-08-25T16:10:33.456Z baf51572-de24-4b32-9421-4610bf831000 Task timed out after 3.00 seconds
2021-08-25T16:10:34.689Z INFO instance/beat.go:665 Home path: [C:\Program Files\functionbeat-7.14.0-windows-x86_64] Config path: [C:\Program Files\functionbeat-7.14.0-windows-x86_64] Data path: [/tmp] Logs path: [/tmp/logs]
2021-08-25T16:10:34.690Z INFO instance/beat.go:673 Beat ID: 4378232e-e2bc-4b36-b84e-30aae048ae34
This is my config (I removed commented out and empty lines for shortness)
functionbeat.provider.aws.endpoint: "s3.amazonaws.com"
functionbeat.provider.aws.deploy_bucket: "events-functionbeat"
functionbeat.provider.aws.region: "eu-west-1"
functionbeat.provider.aws.functions:
- name: cloudwatch
enabled: true
type: cloudwatch_logs
description: "Functionbeat lambda function for uploading cloudwatch logs to logstash"
region: "eu-west-1"
triggers:
- log_group_name: xxx
- name: sqs
enabled: false
type: sqs
description: "lambda function for SQS events"
triggers:
- event_source_arn: arn:aws:sqs:us-east-1:xxxxx:myevents
- name: kinesis
enabled: false
type: kinesis
description: "lambda function for Kinesis events"
triggers:
- event_source_arn: arn:aws:kinesis:us-east-1:xxxxx:myevents
- name: cloudwatch-logs-kinesis
enabled: false
type: cloudwatch_logs_kinesis
description: "lambda function for Cloudwatch logs in Kinesis events"
triggers:
- event_source_arn: arn:aws:kinesis:us-east-1:xxxxx:myevents
setup.template.settings:
index.number_of_shards: 1
setup.kibana:
output.logstash:
hosts: ["x.x.x.x:5044"]
processors:
- add_host_metadata: ~
- add_cloud_metadata: ~
Any help appreciated, thanks.