AWSFargate module for Filebeat not working?

Elastic Stack Beats
,
1

I am attempting to use the AWSFargate module for Filebeat.
Ran the following commands (with placeholder values substituted):

./filebeat.exe modules enable awsfargate

./filebeat -e -E cloud.id=myElasticCloudID -E cloud.auth=elastic:myElasticPassword -M aws.fargate.var.log_group_name=myCloudWatchLogGroupName -M aws.fargate.var.region_name=eu-west-2 -M aws.fargate.var.log_stream_prefix=myLogStreamPrefix -M aws.fargate.var.role_arn=myAWSRoleArn

But it's not working - it connects to Elastic OK but no logs are being sent. Have already successfully set up Metricbeat in a similar way. Have tried with AWS credentials instead of a role and various other things such as using a log_group_arn instead of the group name / region combo and removing the prefix but none of these have made any difference. Was hoping to be able to find the problem from the logs/output but there really isn't much to go on. Grateful for any pointers....

2

Hi!

Please provide what versions you are running and Filbeat's logs.

Pinging @Kaiyan_Sheng here, who originally developed this module.

3

Hi,

Thanks for the quick reply.
I am running version 7.13.3.

Please see logs below with XXX marking obfuscated values.

Enabled awsfargate
2021-07-29T06:26:18.754Z	INFO	instance/beat.go:665	Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs]
2021-07-29T06:26:18.754Z	INFO	instance/beat.go:673	Beat ID: XXX-XXX-XXX-XXX-XXX
2021-07-29T06:26:18.755Z	INFO	[seccomp]	seccomp/seccomp.go:124	Syscall filter successfully installed
2021-07-29T06:26:18.755Z	INFO	[beat]	instance/beat.go:1014	Beat info	
{
    "system_info": {
        "beat": {
            "path": {
                "config": "/usr/share/filebeat",
                "data": "/usr/share/filebeat/data",
                "home": "/usr/share/filebeat",
                "logs": "/usr/share/filebeat/logs"
            },
            "type": "filebeat",
            "uuid": "XXX-XXX-XXX-XXX-XXX"
        }
    }
}
2021-07-29T06:26:18.755Z	INFO	[beat]	instance/beat.go:1023	Build info	
{
    "system_info": {
        "build": {
            "commit": "3ddad4cee7394d1643023604f246cd5ab6d8cfbb",
            "libbeat": "7.13.3",
            "time": "2021-07-02T12:11:38.000Z",
            "version": "7.13.3"
        }
    }
}
2021-07-29T06:26:18.755Z	INFO	[beat]	instance/beat.go:1026	Go runtime info	
{
    "system_info": {
        "go": {
            "os": "linux",
            "arch": "amd64",
            "max_procs": 2,
            "version": "go1.15.13"
        }
    }
}
2021-07-29T06:26:18.757Z	INFO	[beat]	instance/beat.go:1030	Host info	
{
    "system_info": {
        "host": {
            "architecture": "x86_64",
            "boot_time": "2021-07-29T06:23:25Z",
            "containerized": false,
            "name": "XXX-XXX-XXX-XXX-XXX.eu-west-2.compute.internal",
            "ip": [
                "XXX.XXX.XXX.XXX/8",
                "XXX.XXX.XXX.XXX/22",
                "XXX.XXX.XXX.XXX/24"
            ],
            "kernel_version": "4.14.232-176.381.amzn2.x86_64",
            "mac": [
                "0a:58:a9:fe:ac:2a",
                "0a:91:70:2e:6f:5c"
            ],
            "os": {
                "type": "linux",
                "family": "redhat",
                "platform": "centos",
                "name": "CentOS Linux",
                "version": "7 (Core)",
                "major": 7,
                "minor": 9,
                "patch": 2009,
                "codename": "Core"
            },
            "timezone": "UTC",
            "timezone_offset_sec": 0,
            "id": "XXX"
        }
    }
}
2021-07-29T06:26:18.757Z	INFO	[add_cloud_metadata]	add_cloud_metadata/add_cloud_metadata.go:101	add_cloud_metadata: hosting provider type not detected.
2021-07-29T06:26:18.758Z	INFO	[beat]	instance/beat.go:1059	Process info	
{
    "system_info": {
        "process": {
            "capabilities": {
                "inheritable": [
                    "chown",
                    "dac_override",
                    "fowner",
                    "fsetid",
                    "kill",
                    "setgid",
                    "setuid",
                    "setpcap",
                    "net_bind_service",
                    "net_raw",
                    "sys_chroot",
                    "mknod",
                    "audit_write",
                    "setfcap"
                ],
                "permitted": null,
                "effective": null,
                "bounding": [
                    "chown",
                    "dac_override",
                    "fowner",
                    "fsetid",
                    "kill",
                    "setgid",
                    "setuid",
                    "setpcap",
                    "net_bind_service",
                    "net_raw",
                    "sys_chroot",
                    "mknod",
                    "audit_write",
                    "setfcap"
                ],
                "ambient": null
            },
            "cwd": "/usr/share/filebeat",
            "exe": "/usr/share/filebeat/filebeat",
            "name": "filebeat",
            "pid": 15,
            "ppid": 1,
            "seccomp": {
                "mode": "filter",
                "no_new_privs": true
            },
            "start_time": "2021-07-29T06:26:16.900Z"
        }
    }
}
2021-07-29T06:26:18.758Z	INFO	instance/beat.go:309	Setup Beat: filebeat; Version: 7.13.3
2021-07-29T06:26:18.758Z	INFO	[index-management]	idxmgmt/std.go:184	Set output.elasticsearch.index to 'filebeat-7.13.3' as ILM is enabled.
2021-07-29T06:26:18.758Z	INFO	eslegclient/connection.go:99	elasticsearch url: https://925c269adc634093af83e37885006f4e.eu-west-2.ece.gss.gov.uk:443
2021-07-29T06:26:18.759Z	INFO	[publisher]	pipeline/module.go:113	Beat name: XXX-XXX-XXX-XXX-XXX.eu-west-2.compute.internal
2021-07-29T06:26:18.846Z	INFO	instance/beat.go:473	filebeat start running.
2021-07-29T06:26:18.846Z	INFO	[monitoring]	log/log.go:117	Starting metrics logging every 30s
2021-07-29T06:26:18.849Z	INFO	memlog/store.go:119	Loading data file of '/usr/share/filebeat/data/registry/filebeat' succeeded. Active transaction id=0
2021-07-29T06:26:18.849Z	INFO	memlog/store.go:124	Finished loading transaction log file for '/usr/share/filebeat/data/registry/filebeat'. Active transaction id=0
2021-07-29T06:26:18.850Z	INFO	[registrar]	registrar/registrar.go:109	States Loaded from registrar: 0
2021-07-29T06:26:18.850Z	INFO	[crawler]	beater/crawler.go:71	Loading Inputs: 0
2021-07-29T06:26:18.850Z	INFO	[crawler]	beater/crawler.go:108	Loading and starting Inputs completed. Enabled inputs: 0
2021-07-29T06:26:18.850Z	INFO	cfgfile/reload.go:164	Config reloader started
2021-07-29T06:26:18.852Z	INFO	eslegclient/connection.go:99	elasticsearch url: https://925c269adc634093af83e37885006f4e.eu-west-2.ece.gss.gov.uk:443
2021-07-29T06:26:19.365Z	INFO	[esclientleg]	eslegclient/connection.go:314	Attempting to connect to Elasticsearch version 7.12.0
2021-07-29T06:26:19.365Z	INFO	cfgfile/reload.go:224	Loading of config files completed.
2021-07-29T06:26:48.849Z	INFO	[monitoring]	log/log.go:144	Non-zero metrics in the last 30s	
{
    "monitoring": {
        "metrics": {
            "beat": {
                "cgroup": {
                    "cpu": {
                        "cfs": {
                            "period": {
                                "us": 100000
                            }
                        },
                        "id": "XXX"
                    },
                    "cpuacct": {
                        "id": "XXX",
                        "total": {
                            "ns": 735687914
                        }
                    },
                    "memory": {
                        "id": "XXX",
                        "mem": {
                            "limit": {
                                "bytes": 9223372036854771712
                            },
                            "usage": {
                                "bytes": 54972416
                            }
                        }
                    }
                },
                "cpu": {
                    "system": {
                        "ticks": 30,
                        "time": {
                            "ms": 36
                        }
                    },
                    "total": {
                        "ticks": 250,
                        "time": {
                            "ms": 260
                        },
                        "value": 0
                    },
                    "user": {
                        "ticks": 220,
                        "time": {
                            "ms": 224
                        }
                    }
                },
                "handles": {
                    "limit": {
                        "hard": 4096,
                        "soft": 1024
                    },
                    "open": 10
                },
                "info": {
                    "ephemeral_id": "XXX-XXX-XXX-XXX-XXX",
                    "uptime": {
                        "ms": 30893
                    }
                },
                "memstats": {
                    "gc_next": 22675472,
                    "memory_alloc": 12511264,
                    "memory_sys": 75318272,
                    "memory_total": 50894424,
                    "rss": 123715584
                },
                "runtime": {
                    "goroutines": 20
                }
            },
            "filebeat": {
                "harvester": {
                    "open_files": 0,
                    "running": 0
                }
            },
            "libbeat": {
                "config": {
                    "module": {
                        "running": 1,
                        "starts": 1
                    },
                    "reloads": 1,
                    "scans": 1
                },
                "output": {
                    "events": {
                        "active": 0
                    },
                    "type": "elasticsearch"
                },
                "pipeline": {
                    "clients": 0,
                    "events": {
                        "active": 0
                    },
                    "queue": {
                        "max_events": 4096
                    }
                }
            },
            "registrar": {
                "states": {
                    "current": 0
                }
            },
            "system": {
                "cpu": {
                    "cores": 2
                },
                "load": {
                    "1": 0.76,
                    "5": 0.27,
                    "15": 0.09,
                    "norm": {
                        "1": 0.38,
                        "5": 0.135,
                        "15": 0.045
                    }
                }
            }
        }
    }
}
2021-07-29T06:27:18.848Z	INFO	[monitoring]	log/log.go:144	Non-zero metrics in the last 30s	
{
    "monitoring": {
        "metrics": {
            "beat": {
                "cgroup": {
                    "cpuacct": {
                        "total": {
                            "ns": 5399621
                        }
                    },
                    "memory": {
                        "mem": {
                            "usage": {
                                "bytes": 139264
                            }
                        }
                    }
                },
                "cpu": {
                    "system": {
                        "ticks": 30,
                        "time": {
                            "ms": 1
                        }
                    },
                    "total": {
                        "ticks": 260,
                        "time": {
                            "ms": 7
                        },
                        "value": 260
                    },
                    "user": {
                        "ticks": 230,
                        "time": {
                            "ms": 6
                        }
                    }
                },
                "handles": {
                    "limit": {
                        "hard": 4096,
                        "soft": 1024
                    },
                    "open": 10
                },
                "info": {
                    "ephemeral_id": "XXX-XXX-XXX-XXX-XXX",
                    "uptime": {
                        "ms": 60891
                    }
                },
                "memstats": {
                    "gc_next": 22675472,
                    "memory_alloc": 12665240,
                    "memory_total": 51048400,
                    "rss": 123715584
                },
                "runtime": {
                    "goroutines": 20
                }
            },
            "filebeat": {
                "harvester": {
                    "open_files": 0,
                    "running": 0
                }
            },
            "libbeat": {
                "config": {
                    "module": {
                        "running": 1
                    }
                },
                "output": {
                    "events": {
                        "active": 0
                    }
                },
                "pipeline": {
                    "clients": 0,
                    "events": {
                        "active": 0
                    }
                }
            },
            "registrar": {
                "states": {
                    "current": 0
                }
            },
            "system": {
                "load": {
                    "1": 0.46,
                    "5": 0.25,
                    "15": 0.09,
                    "norm": {
                        "1": 0.23,
                        "5": 0.125,
                        "15": 0.045
                    }
                }
            }
        }
    }
}
4

One point of confusion here is exactly what field values to supply.
The documentation has this example config:

- module: aws
  fargate:
    enabled: true
    var.credential_profile_name: test-filebeat
    var.log_group_arn: arn:aws:logs:us-east-1:1234567890:log-group:/ecs/test-log-group:*

But the original Pull Request had:

- module: awsfargate
  log:
    enabled: true
    var.credential_profile_name: elastic-beats
    var.log_group_arn: arn:aws:logs:us-east-1:1234567890:log-group:/ecs/metricbeat-awsfargate:*

Note: These are different - one starts aws -> fargate and the other starts awsfargate -> log. Are both needed or does one supersede the other? Have tried with one, the other and both but still not got it working - here is an example alternative command I tried:

./filebeat -e -E cloud.id=myElasticCloudID -E cloud.auth=elastic:myElasticPassword -M awsfargate.log.enabled=myCloudWatchLogGroupName -M awsfargate.log.var.log_group_arn=myLogGroupArn -M awsfargate.log.var.role_arn=myAWSRoleArn

@Kaiyan_Sheng / @ChrsMark Any help you could provide with the above (or even just an update to say it's being looked into) would be much appreciated.

5

Hi @stec00 , thanks for posting here and sorry for the confusion. The correct config is the second one:

- module: awsfargate
  log:
    enabled: true
    var.credential_profile_name: elastic-beats
    var.log_group_arn: arn:aws:logs:us-east-1:1234567890:log-group:/ecs/metricbeat-awsfargate:*

I would give this config in aws.yml a try, and then start Filebeat with ./filebeat -e -E cloud.id=myElasticCloudID -E cloud.auth=elastic:myElasticPassword -d "*". Then you should be able to see all the debug level logs, that might have more information there. Thanks!