Filebeat - Multiple VPC logs from different S3 buckets not working

Elastic Stack Beats
1

HI All,

Within our AWS account, we have 2 VPC's that output flow logs into individual separate S3 buckets. I have enabled event notifications and connected them to SQS which all works. I also have WAF and ALB logs which I have done the same.

However I can't get filebeat to pull both sets of logs, I can only get one or the other.

Following this article I assumed you could put 2 AWS modules and 2 filebeat inputs into the config but it only seems to process the last element in the config. In the case below I get everything from queues starting dev- but nothing from queues starting prod-. If I change the order the reverse happens.

Can anyone help?

filebeat.inputs:
- type: aws-s3
  queue_url: "prod-waf-queue-url-placeholder"
  role_arn: '${ROLE_ARN}'
  content_type: application/json
- type: aws-s3
  queue_url: "dev-waf-queue-url-placeholder"
  role_arn: '${ROLE_ARN}'
  content_type: application/json

filebeat.modules:
- module: aws
  elb:
    enabled: true
    var.queue_url: "prod-elb-queue-url-placeholder"
    var.role_arn: '${ROLE_ARN}'
  vpcflow:
    enabled: true
    var.queue_url: "prod-vpc-queue-url-placeholder"
    var.role_arn: '${ROLE_ARN}'
- module: aws
  elb:
    enabled: true
    var.queue_url: "dev-elb-queue-url-placeholder"
    var.role_arn: '${ROLE_ARN}'
  vpcflow:
    enabled: true
    var.queue_url: "dev-vpc-queue-url-placeholder"
    var.role_arn: '${ROLE_ARN}'


output.logstash:
  hosts: ['${LOGSTASH_URL:?No logstash URL detected. Please set env variable LOGSTASH_URL}:5044']
2

What you have should work. Is filebeat outputting anything in it's logs?

3

Hi,

Nothing untoward. There are some warnings that neither queue URL or bucket arn were provided but I am assuming these are from the other modules?

	Enabled modules/filesets: aws (cloudtrail, cloudwatch, ec2, elb, s3access, vpcflow),  ()"
1640623375592,2021-12-27T16:42:55.592Z	WARN	beater/filebeat.go:136	Fileset `cloudtrail` for module `aws` is loaded but was not explicitly defined in the config. Starting from v8.0 this fileset won't be loaded unless explicitly defined.
1640623375592,2021-12-27T16:42:55.592Z	WARN	beater/filebeat.go:136	Fileset `cloudwatch` for module `aws` is loaded but was not explicitly defined in the config. Starting from v8.0 this fileset won't be loaded unless explicitly defined.
1640623375593,2021-12-27T16:42:55.592Z	WARN	beater/filebeat.go:136	Fileset `ec2` for module `aws` is loaded but was not explicitly defined in the config. Starting from v8.0 this fileset won't be loaded unless explicitly defined.
1640623375593,2021-12-27T16:42:55.593Z	WARN	beater/filebeat.go:136	Fileset `s3access` for module `aws` is loaded but was not explicitly defined in the config. Starting from v8.0 this fileset won't be loaded unless explicitly defined.
1640623375595,"2021-12-27T16:42:55.595Z	WARN	beater/filebeat.go:202	Filebeat is unable to load the ingest pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the ingest pipelines or are using Logstash pipelines, you can ignore this warning."
1640623375673,2021-12-27T16:42:55.672Z	INFO	instance/beat.go:492	filebeat start running.
1640623375673,2021-12-27T16:42:55.672Z	INFO	[monitoring]	log/log.go:142	Starting metrics logging every 30s
1640623375673,"2021-12-27T16:42:55.672Z	WARN	beater/filebeat.go:231	Filebeat is unable to load the ingest pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the ingest pipelines or are using Logstash pipelines, you can ignore this warning."
1640623375675,2021-12-27T16:42:55.675Z	INFO	memlog/store.go:119	Loading data file of '/usr/share/filebeat/data/registry/filebeat' succeeded. Active transaction id=0
1640623375675,2021-12-27T16:42:55.675Z	INFO	memlog/store.go:124	Finished loading transaction log file for '/usr/share/filebeat/data/registry/filebeat'. Active transaction id=0
1640623375675,"2021-12-27T16:42:55.675Z	WARN	beater/filebeat.go:411	Filebeat is unable to load the ingest pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the ingest pipelines or are using Logstash pipelines, you can ignore this warning."
1640623375675,2021-12-27T16:42:55.675Z	INFO	[registrar]	registrar/registrar.go:109	States Loaded from registrar: 0
1640623375680,2021-12-27T16:42:55.675Z	INFO	[crawler]	beater/crawler.go:71	Loading Inputs: 8
1640623375680,2021-12-27T16:42:55.676Z	INFO	[crawler]	beater/crawler.go:141	Starting input (ID: 16607647092026520318)
1640623375680,2021-12-27T16:42:55.676Z	INFO	[crawler]	beater/crawler.go:141	Starting input (ID: 6799548206363597317)
1640623375680,"2021-12-27T16:42:55.676Z	WARN	[aws-s3]	awss3/config.go:58	Neither queue_url nor bucket_arn were provided, input aws-s3 will stop.Starting from v8.0 this will prevent Filebeat to start."
1640623375680,2021-12-27T16:42:55.676Z	INFO	[crawler]	beater/crawler.go:141	Starting input (ID: 1614679342181170108)
1640623375680,2021-12-27T16:42:55.676Z	INFO	[crawler]	beater/crawler.go:141	Starting input (ID: 12161307968425761396)
1640623375680,"2021-12-27T16:42:55.676Z	WARN	[aws-s3]	awss3/config.go:58	Neither queue_url nor bucket_arn were provided, input aws-s3 will stop.Starting from v8.0 this will prevent Filebeat to start."
1640623375680,2021-12-27T16:42:55.676Z	INFO	[crawler]	beater/crawler.go:141	Starting input (ID: 1199955525200915081)
1640623375680,2021-12-27T16:42:55.677Z	INFO	[crawler]	beater/crawler.go:141	Starting input (ID: 6814296712940442267)
1640623375680,"2021-12-27T16:42:55.677Z	WARN	[aws-s3]	awss3/config.go:58	Neither queue_url nor bucket_arn were provided, input aws-s3 will stop.Starting from v8.0 this will prevent Filebeat to start."
1640623375680,2021-12-27T16:42:55.677Z	INFO	[crawler]	beater/crawler.go:141	Starting input (ID: 15767413901316835177)
1640623375680,"2021-12-27T16:42:55.677Z	WARN	[aws-s3]	awss3/config.go:58	Neither queue_url nor bucket_arn were provided, input aws-s3 will stop.Starting from v8.0 this will prevent Filebeat to start."
1640623375680,2021-12-27T16:42:55.677Z	INFO	[crawler]	beater/crawler.go:141	Starting input (ID: 360984977805401220)
1640623375680,2021-12-27T16:42:55.677Z	INFO	[crawler]	beater/crawler.go:108	Loading and starting Inputs completed. Enabled inputs: 8
1640623375680,"2021-12-27T16:42:55.677Z	INFO	[input.aws-s3]	compat/compat.go:111	Input aws-s3 starting	{""id"": ""E67A370A034A32FE""}"

Then when it gets down to me seeing the sqs queues starting I normally see a start message with an ID then info about setting the region etc.

But I also then just get the following with no other error messages mentioning that ID.

1640623375681,"2021-12-27T16:42:55.678Z	INFO	[input.aws-s3]	compat/compat.go:111	Input aws-s3 starting	{""id"": "" 50279ECBB57DC84""}"
1640623375681,"2021-12-27T16:42:55.678Z	INFO	[input.aws-s3]	compat/compat.go:124	Input 'aws-s3' stopped	{""id"": "" 50279ECBB57DC84""}"
4

First thing I would do is add the other AWS filesets and explicitly disable them so to remove a bunch of the extra logs l. Then try again and let's see what shows up in the logs next.

5

So yeah that got rid of all the errors. The following is my log output with bits redacted but it just completely disregards the first AWS module defined in the config:

timestamp,message
1640627035122,2021-12-27T17:43:55.122Z	INFO	instance/beat.go:686	Home path: [/usr/share/filebeat] Config path: [/usr/share/filebeat] Data path: [/usr/share/filebeat/data] Logs path: [/usr/share/filebeat/logs] Hostfs Path: [/]
1640627035130,2021-12-27T17:43:55.130Z	INFO	instance/beat.go:694	Beat ID: 995f62e2-a24a-4ce2-99c3-62b164ab37ad
1640627035130,2021-12-27T17:43:55.130Z	INFO	[seccomp]	seccomp/seccomp.go:124	Syscall filter successfully installed
1640627035130,"2021-12-27T17:43:55.130Z	INFO	[beat]	instance/beat.go:1040	Beat info	{""system_info"": {""beat"": {""path"": {""config"": ""/usr/share/filebeat"", ""data"": ""/usr/share/filebeat/data"", ""home"": ""/usr/share/filebeat"", ""logs"": ""/usr/share/filebeat/logs""}, ""type"": ""filebeat"", ""uuid"": ""995f62e2-a24a-4ce2-99c3-62b164ab37ad""}}}"
1640627035130,"2021-12-27T17:43:55.130Z	INFO	[beat]	instance/beat.go:1049	Build info	{""system_info"": {""build"": {""commit"": ""7e56c4a053a2fe26c0cac168dd974780428a2aa6"", ""libbeat"": ""7.16.1"", ""time"": ""2021-12-11T01:49:16.000Z"", ""version"": ""7.16.1""}}}"
1640627035130,"2021-12-27T17:43:55.130Z	INFO	[beat]	instance/beat.go:1052	Go runtime info	{""system_info"": {""go"": {""os"":""linux"",""arch"":""amd64"",""max_procs"":2,""version"":""go1.17.2""}}}"
1640627035131,"2021-12-27T17:43:55.131Z	INFO	[beat]	instance/beat.go:1056	Host info	{""system_info"": {""host"": {""architecture"":""x86_64"",""boot_time"":""2021-12-27T17:40:41Z"",""containerized"":false,""name"":""*removed*"",""ip"":[""127.0.0.1/8"",""::1/128"",""169.254.172.2/22"",""fe80::2c65:83ff:feac:7d11/64"",""*internalip*"",""fe80::825:59ff:fe9f:da58/64""],""kernel_version"":""4.14.252-195.483.amzn2.x86_64"",""mac"":[""0a:58:a9:fe:ac:02"",""0a:25:59:9f:da:58""],""os"":{""type"":""linux"",""family"":""redhat"",""platform"":""centos"",""name"":""CentOS Linux"",""version"":""7 (Core)"",""major"":7,""minor"":9,""patch"":2009,""codename"":""Core""},""timezone"":""UTC"",""timezone_offset_sec"":0,""id"":""9fcaada5ef8a36e0af98f82e5022a01f""}}}"
1640627035131,"2021-12-27T17:43:55.131Z	INFO	[beat]	instance/beat.go:1085	Process info	{""system_info"": {""process"": {""capabilities"": {""inheritable"":[""chown"",""dac_override"",""fowner"",""fsetid"",""kill"",""setgid"",""setuid"",""setpcap"",""net_bind_service"",""net_raw"",""sys_chroot"",""mknod"",""audit_write"",""setfcap""],""permitted"":null,""effective"":null,""bounding"":[""chown"",""dac_override"",""fowner"",""fsetid"",""kill"",""setgid"",""setuid"",""setpcap"",""net_bind_service"",""net_raw"",""sys_chroot"",""mknod"",""audit_write"",""setfcap""],""ambient"":null}, ""cwd"": ""/usr/share/filebeat"", ""exe"": ""/usr/share/filebeat/filebeat"", ""name"": ""filebeat"", ""pid"": 10, ""ppid"": 1, ""seccomp"": {""mode"":""filter"",""no_new_privs"":true}, ""start_time"": ""2021-12-27T17:43:54.130Z""}}}"
1640627035131,2021-12-27T17:43:55.131Z	INFO	instance/beat.go:328	Setup Beat: filebeat; Version: 7.16.1
1640627035132,2021-12-27T17:43:55.132Z	INFO	[publisher]	pipeline/module.go:113	Beat name: *removed*
1640627035135,"2021-12-27T17:43:55.135Z	INFO	beater/filebeat.go:118	Enabled modules/filesets: aws (elb, vpcflow),  ()"
1640627035136,"2021-12-27T17:43:55.136Z	WARN	beater/filebeat.go:202	Filebeat is unable to load the ingest pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the ingest pipelines or are using Logstash pipelines, you can ignore this warning."
1640627035136,2021-12-27T17:43:55.136Z	INFO	[monitoring]	log/log.go:142	Starting metrics logging every 30s
1640627035136,2021-12-27T17:43:55.136Z	INFO	instance/beat.go:492	filebeat start running.
1640627035136,"2021-12-27T17:43:55.136Z	WARN	beater/filebeat.go:231	Filebeat is unable to load the ingest pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the ingest pipelines or are using Logstash pipelines, you can ignore this warning."
1640627035139,2021-12-27T17:43:55.139Z	INFO	memlog/store.go:119	Loading data file of '/usr/share/filebeat/data/registry/filebeat' succeeded. Active transaction id=0
1640627035139,2021-12-27T17:43:55.139Z	INFO	memlog/store.go:124	Finished loading transaction log file for '/usr/share/filebeat/data/registry/filebeat'. Active transaction id=0
1640627035139,"2021-12-27T17:43:55.139Z	WARN	beater/filebeat.go:411	Filebeat is unable to load the ingest pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the ingest pipelines or are using Logstash pipelines, you can ignore this warning."
1640627035141,2021-12-27T17:43:55.139Z	INFO	[registrar]	registrar/registrar.go:109	States Loaded from registrar: 0
1640627035141,2021-12-27T17:43:55.139Z	INFO	[crawler]	beater/crawler.go:71	Loading Inputs: 4
1640627035141,2021-12-27T17:43:55.139Z	INFO	[crawler]	beater/crawler.go:141	Starting input (ID: 16607647092026520318)
1640627035141,"2021-12-27T17:43:55.140Z	INFO	[input.aws-s3]	compat/compat.go:111	Input aws-s3 starting	{""id"": ""E67A370A034A32FE""}"
1640627035142,"2021-12-27T17:43:55.140Z	INFO	[input.aws-s3]	awss3/input.go:176	AWS api_timeout is set to 2m0s.	{""id"": ""E67A370A034A32FE"", ""queue_url"": ""*prod-waf-logs-queue-url*""}"
1640627035142,"2021-12-27T17:43:55.140Z	INFO	[input.aws-s3]	awss3/input.go:177	AWS region is set to *region*.	{""id"": ""E67A370A034A32FE"", ""queue_url"": ""*prod-waf-logs-queue-url*""}"
1640627035142,"2021-12-27T17:43:55.140Z	INFO	[input.aws-s3]	awss3/input.go:178	AWS SQS visibility_timeout is set to 5m0s.	{""id"": ""E67A370A034A32FE"", ""queue_url"": ""*prod-waf-logs-queue-url*""}"
1640627035142,"2021-12-27T17:43:55.140Z	INFO	[input.aws-s3]	awss3/input.go:179	AWS SQS max_number_of_messages is set to 5.	{""id"": ""E67A370A034A32FE"", ""queue_url"": ""*prod-waf-logs-queue-url*""}"
1640627035142,2021-12-27T17:43:55.140Z	INFO	[crawler]	beater/crawler.go:141	Starting input (ID: 6799548206363597317)
1640627035142,2021-12-27T17:43:55.140Z	INFO	[crawler]	beater/crawler.go:141	Starting input (ID: 12161307968425761396)
1640627035142,"2021-12-27T17:43:55.140Z	INFO	[input.aws-s3]	compat/compat.go:111	Input aws-s3 starting	{""id"": ""5E5CD91171725A05""}"
1640627035142,2021-12-27T17:43:55.141Z	INFO	[crawler]	beater/crawler.go:141	Starting input (ID: 6814296712940442267)
1640627035142,"2021-12-27T17:43:55.141Z	INFO	[input.aws-s3]	awss3/input.go:176	AWS api_timeout is set to 2m0s.	{""id"": ""5E5CD91171725A05"", ""queue_url"": ""*dev-waf-logs-queue-url*""}"
1640627035142,"2021-12-27T17:43:55.141Z	INFO	[input.aws-s3]	awss3/input.go:177	AWS region is set to *region*.	{""id"": ""5E5CD91171725A05"", ""queue_url"": ""*dev-waf-logs-queue-url*""}"
1640627035142,"2021-12-27T17:43:55.141Z	INFO	[input.aws-s3]	compat/compat.go:111	Input aws-s3 starting	{""id"": ""5E913EC1B8ED529B""}"
1640627035142,"2021-12-27T17:43:55.141Z	INFO	[input.aws-s3]	awss3/input.go:178	AWS SQS visibility_timeout is set to 5m0s.	{""id"": ""5E5CD91171725A05"", ""queue_url"": ""*dev-waf-logs-queue-url*""}"
1640627035142,"2021-12-27T17:43:55.141Z	INFO	[input.aws-s3]	awss3/input.go:179	AWS SQS max_number_of_messages is set to 5.	{""id"": ""5E5CD91171725A05"", ""queue_url"": ""*dev-waf-logs-queue-url*""}"
1640627035142,"2021-12-27T17:43:55.141Z	INFO	[input.aws-s3]	awss3/input.go:176	AWS api_timeout is set to 2m0s.	{""id"": ""5E913EC1B8ED529B"", ""queue_url"": ""*dev-vpc-logs-queue-url*""}"
1640627035142,"2021-12-27T17:43:55.141Z	INFO	[input.aws-s3]	awss3/input.go:177	AWS region is set to *region*.	{""id"": ""5E913EC1B8ED529B"", ""queue_url"": ""*dev-vpc-logs-queue-url*""}"
1640627035142,"2021-12-27T17:43:55.141Z	INFO	[input.aws-s3]	awss3/input.go:178	AWS SQS visibility_timeout is set to 5m0s.	{""id"": ""5E913EC1B8ED529B"", ""queue_url"": ""*dev-vpc-logs-queue-url*""}"
1640627035142,"2021-12-27T17:43:55.141Z	INFO	[input.aws-s3]	awss3/input.go:179	AWS SQS max_number_of_messages is set to 5.	{""id"": ""5E913EC1B8ED529B"", ""queue_url"": ""*dev-vpc-logs-queue-url*""}"
1640627035142,2021-12-27T17:43:55.141Z	INFO	[crawler]	beater/crawler.go:108	Loading and starting Inputs completed. Enabled inputs: 4
1640627035142,"2021-12-27T17:43:55.141Z	INFO	[input.aws-s3]	compat/compat.go:111	Input aws-s3 starting	{""id"": ""A8C5A529F6251A74""}"
1640627035142,"2021-12-27T17:43:55.141Z	INFO	[input.aws-s3]	awss3/input.go:176	AWS api_timeout is set to 2m0s.	{""id"": ""A8C5A529F6251A74"", ""queue_url"": ""*dev-alb-logs-queue-url*""}"
1640627035142,"2021-12-27T17:43:55.141Z	INFO	[input.aws-s3]	awss3/input.go:177	AWS region is set to *region*.	{""id"": ""A8C5A529F6251A74"", ""queue_url"": ""*dev-alb-logs-queue-url*""}"
1640627035142,"2021-12-27T17:43:55.141Z	INFO	[input.aws-s3]	awss3/input.go:178	AWS SQS visibility_timeout is set to 5m0s.	{""id"": ""A8C5A529F6251A74"", ""queue_url"": ""*dev-alb-logs-queue-url*""}"
1640627035142,"2021-12-27T17:43:55.141Z	INFO	[input.aws-s3]	awss3/input.go:179	AWS SQS max_number_of_messages is set to 5.	{""id"": ""A8C5A529F6251A74"", ""queue_url"": ""*dev-alb-logs-queue-url*""}"
1640627065143,"2021-12-27T17:44:25.143Z	INFO	[monitoring]	log/log.go:184	Non-zero metrics in the last 30s	{""monitoring"": {""metrics"": {""beat"":{""cgroup"":{""cpu"":{""cfs"":{""period"":{""us"":100000}},""id"":""/""},""cpuacct"":{""id"":""/"",""total"":{""ns"":418423008}},""memory"":{""id"":""/"",""mem"":{""limit"":{""bytes"":536870912},""usage"":{""bytes"":53551104}}}},""cpu"":{""system"":{""ticks"":50,""time"":{""ms"":55}},""total"":{""ticks"":200,""time"":{""ms"":217},""value"":200},""user"":{""ticks"":150,""time"":{""ms"":162}}},""handles"":{""limit"":{""hard"":4096,""soft"":1024},""open"":17},""info"":{""ephemeral_id"":""ca6fdfa1-29eb-4833-9071-af4bd109691b"",""uptime"":{""ms"":30421},""version"":""7.16.1""},""memstats"":{""gc_next"":25565408,""memory_alloc"":21530808,""memory_sys"":41501704,""memory_total"":62419096,""rss"":128552960},""runtime"":{""goroutines"":48}},""filebeat"":{""harvester"":{""open_files"":0,""running"":0}},""libbeat"":{""config"":{""module"":{""running"":0}},""output"":{""events"":{""active"":0},""type"":""logstash""},""pipeline"":{""clients"":4,""events"":{""active"":0},""queue"":{""max_events"":4096}}},""registrar"":{""states"":{""current"":0}},""system"":{""cpu"":{""cores"":2},""load"":{""1"":0.2,""15"":0.06,""5"":0.13,""norm"":{""1"":0.1,""15"":0.03,""5"":0.065}}}}}}"
1640627095142,"2021-12-27T17:44:55.141Z	INFO	[monitoring]	log/log.go:184	Non-zero metrics in the last 30s	{""monitoring"": {""metrics"": {""beat"":{""cgroup"":{""cpuacct"":{""total"":{""ns"":24912343}},""memory"":{""mem"":{""usage"":{""bytes"":-3579904}}}},""cpu"":{""system"":{""ticks"":50,""time"":{""ms"":3}},""total"":{""ticks"":230,""time"":{""ms"":24},""value"":230},""user"":{""ticks"":180,""time"":{""ms"":21}}},""handles"":{""limit"":{""hard"":4096,""soft"":1024},""open"":17},""info"":{""ephemeral_id"":""ca6fdfa1-29eb-4833-9071-af4bd109691b"",""uptime"":{""ms"":60414},""version"":""7.16.1""},""memstats"":{""gc_next"":21874528,""memory_alloc"":11208768,""memory_sys"":262144,""memory_total"":63344408,""rss"":125206528},""runtime"":{""goroutines"":48}},""filebeat"":{""harvester"":{""open_files"":0,""running"":0}},""libbeat"":{""config"":{""module"":{""running"":0}},""output"":{""events"":{""active"":0}},""pipeline"":{""clients"":4,""events"":{""active"":0}}},""registrar"":{""states"":{""current"":0}},""system"":{""load"":{""1"":0.12,""15"":0.05,""5"":0.12,""norm"":{""1"":0.06,""15"":0.025,""5"":0.06}}}}}}"
1640627125142,"2021-12-27T17:45:25.141Z	INFO	[monitoring]	log/log.go:184	Non-zero metrics in the last 30s	{""monitoring"": {""metrics"": {""beat"":{""cgroup"":{""cpuacct"":{""total"":{""ns"":14134194}},""memory"":{""mem"":{""usage"":{""bytes"":90112}}}},""cpu"":{""system"":{""ticks"":60,""time"":{""ms"":10}},""total"":{""ticks"":240,""time"":{""ms"":13},""value"":240},""user"":{""ticks"":180,""time"":{""ms"":3}}},""handles"":{""limit"":{""hard"":4096,""soft"":1024},""open"":13},""info"":{""ephemeral_id"":""ca6fdfa1-29eb-4833-9071-af4bd109691b"",""uptime"":{""ms"":90415},""version"":""7.16.1""},""memstats"":{""gc_next"":21874528,""memory_alloc"":12374336,""memory_total"":64509976,""rss"":125206528},""runtime"":{""goroutines"":40}},""filebeat"":{""harvester"":{""open_files"":0,""running"":0}},""libbeat"":{""config"":{""module"":{""running"":0}},""output"":{""events"":{""active"":0}},""pipeline"":{""clients"":4,""events"":{""active"":0}}},""registrar"":{""states"":{""current"":0}},""system"":{""load"":{""1"":0.07,""15"":0.05,""5"":0.11,""norm"":{""1"":0.035,""15"":0.025,""5"":0.055}}}}}}"
1640627155142,"2021-12-27T17:45:55.141Z	INFO	[monitoring]	log/log.go:184	Non-zero metrics in the last 30s	{""monitoring"": {""metrics"": {""beat"":{""cgroup"":{""cpuacct"":{""total"":{""ns"":10355367}},""memory"":{""mem"":{""usage"":{""bytes"":32768}}}},""cpu"":{""system"":{""ticks"":70,""time"":{""ms"":6}},""total"":{""ticks"":260,""time"":{""ms"":10},""value"":260},""user"":{""ticks"":190,""time"":{""ms"":4}}},""handles"":{""limit"":{""hard"":4096,""soft"":1024},""open"":13},""info"":{""ephemeral_id"":""ca6fdfa1-29eb-4833-9071-af4bd109691b"",""uptime"":{""ms"":120416},""version"":""7.16.1""},""memstats"":{""gc_next"":21874528,""memory_alloc"":13478416,""memory_total"":65614056,""rss"":125206528},""runtime"":{""goroutines"":40}},""filebeat"":{""harvester"":{""open_files"":0,""running"":0}},""libbeat"":{""config"":{""module"":{""running"":0}},""output"":{""events"":{""active"":0}},""pipeline"":{""clients"":4,""events"":{""active"":0}}},""registrar"":{""states"":{""current"":0}},""system"":{""load"":{""1"":0.04,""15"":0.05,""5"":0.1,""norm"":{""1"":0.02,""15"":0.025,""5"":0.05}}}}}}"
1640627185142,"2021-12-27T17:46:25.142Z	INFO	[monitoring]	log/log.go:184	Non-zero metrics in the last 30s	{""monitoring"": {""metrics"": {""beat"":{""cgroup"":{""cpuacct"":{""total"":{""ns"":12136907}},""memory"":{""mem"":{""usage"":{""bytes"":28672}}}},""cpu"":{""system"":{""ticks"":70,""time"":{""ms"":5}},""total"":{""ticks"":260,""time"":{""ms"":12},""value"":260},""user"":{""ticks"":190,""time"":{""ms"":7}}},""handles"":{""limit"":{""hard"":4096,""soft"":1024},""open"":13},""info"":{""ephemeral_id"":""ca6fdfa1-29eb-4833-9071-af4bd109691b"",""uptime"":{""ms"":150416},""version"":""7.16.1""},""memstats"":{""gc_next"":21874528,""memory_alloc"":14647008,""memory_total"":66782648,""rss"":125206528},""runtime"":{""goroutines"":40}},""filebeat"":{""harvester"":{""open_files"":0,""running"":0}},""libbeat"":{""config"":{""module"":{""running"":0}},""output"":{""events"":{""active"":0}},""pipeline"":{""clients"":4,""events"":{""active"":0}}},""registrar"":{""states"":{""current"":0}},""system"":{""load"":{""1"":0.02,""15"":0.04,""5"":0.08,""norm"":{""1"":0.01,""15"":0.02,""5"":0.04}}}}}}"
1640627215141,"2021-12-27T17:46:55.141Z	INFO	[monitoring]	log/log.go:184	Non-zero metrics in the last 30s	{""monitoring"": {""metrics"": {""beat"":{""cgroup"":{""cpuacct"":{""total"":{""ns"":34977429}},""memory"":{""mem"":{""usage"":{""bytes"":-3268608}}}},""cpu"":{""system"":{""ticks"":90,""time"":{""ms"":17}},""total"":{""ticks"":300,""time"":{""ms"":34},""value"":300},""user"":{""ticks"":210,""time"":{""ms"":17}}},""handles"":{""limit"":{""hard"":4096,""soft"":1024},""open"":13},""info"":{""ephemeral_id"":""ca6fdfa1-29eb-4833-9071-af4bd109691b"",""uptime"":{""ms"":180415},""version"":""7.16.1""},""memstats"":{""gc_next"":20245424,""memory_alloc"":10399504,""memory_total"":67522464,""rss"":122077184},""runtime"":{""goroutines"":40}},""filebeat"":{""harvester"":{""open_files"":0,""running"":0}},""libbeat"":{""config"":{""module"":{""running"":0}},""output"":{""events"":{""active"":0}},""pipeline"":{""clients"":4,""events"":{""active"":0}}},""registrar"":{""states"":{""current"":0}},""system"":{""load"":{""1"":0.01,""15"":0.04,""5"":0.08,""norm"":{""1"":0.005,""15"":0.02,""5"":0.04}}}}}}"
1640627245141,"2021-12-27T17:47:25.141Z	INFO	[monitoring]	log/log.go:184	Non-zero metrics in the last 30s	{""monitoring"": {""metrics"": {""beat"":{""cgroup"":{""cpuacct"":{""total"":{""ns"":11485105}},""memory"":{""mem"":{""usage"":{""bytes"":-118784}}}},""cpu"":{""system"":{""ticks"":100,""time"":{""ms"":9}},""total"":{""ticks"":310,""time"":{""ms"":12},""value"":310},""user"":{""ticks"":210,""time"":{""ms"":3}}},""handles"":{""limit"":{""hard"":4096,""soft"":1024},""open"":13},""info"":{""ephemeral_id"":""ca6fdfa1-29eb-4833-9071-af4bd109691b"",""uptime"":{""ms"":210415},""version"":""7.16.1""},""memstats"":{""gc_next"":20245424,""memory_alloc"":11554360,""memory_total"":68677320,""rss"":122077184},""runtime"":{""goroutines"":40}},""filebeat"":{""harvester"":{""open_files"":0,""running"":0}},""libbeat"":{""config"":{""module"":{""running"":0}},""output"":{""events"":{""active"":0}},""pipeline"":{""clients"":4,""events"":{""active"":0}}},""registrar"":{""states"":{""current"":0}},""system"":{""load"":{""1"":0.01,""15"":0.04,""5"":0.07,""norm"":{""1"":0.005,""15"":0.02,""5"":0.035}}}}}}"
1640627257140,2021-12-27T17:47:37.140Z	INFO	[publisher_pipeline_output]	pipeline/output.go:143	Connecting to backoff(async(tcp://*logstashurl*))
1640627257220,2021-12-27T17:47:37.220Z	INFO	[publisher]	pipeline/retry.go:219	retryer: send unwait signal to consumer
1640627257220,2021-12-27T17:47:37.220Z	INFO	[publisher]	pipeline/retry.go:223	  done
1640627257223,2021-12-27T17:47:37.223Z	INFO	[publisher_pipeline_output]	pipeline/output.go:151	Connection to backoff(async(tcp://*logstashurl*)) established
1640627275140,"2021-12-27T17:47:55.140Z	INFO	[monitoring]	log/log.go:184	Non-zero metrics in the last 30s	{""monitoring"": {""metrics"": {""beat"":{""cgroup"":{""cpuacct"":{""total"":{""ns"":326048740}},""memory"":{""mem"":{""usage"":{""bytes"":27443200}}}},""cpu"":{""system"":{""ticks"":120,""time"":{""ms"":17}},""total"":{""ticks"":640,""time"":{""ms"":325},""value"":640},""user"":{""ticks"":520,""time"":{""ms"":308}}},""handles"":{""limit"":{""hard"":4096,""soft"":1024},""open"":19},""info"":{""ephemeral_id"":""ca6fdfa1-29eb-4833-9071-af4bd109691b"",""uptime"":{""ms"":240415},""version"":""7.16.1""},""memstats"":{""gc_next"":42357376,""memory_alloc"":36160848,""memory_sys"":13107200,""memory_total"":124340152,""rss"":149295104},""runtime"":{""goroutines"":64}},""filebeat"":{""events"":{""active"":4121,""added"":6169,""done"":2048},""harvester"":{""open_files"":0,""running"":0}},""libbeat"":{""config"":{""module"":{""running"":0}},""output"":{""events"":{""acked"":2048,""active"":4096,""batches"":3,""total"":6144},""read"":{""bytes"":24},""write"":{""bytes"":436089}},""pipeline"":{""clients"":4,""events"":{""active"":4118,""published"":6164,""retry"":2048,""total"":6166},""queue"":{""acked"":2048}}},""registrar"":{""states"":{""current"":0}},""system"":{""load"":{""1"":0,""15"":0.04,""5"":0.06,""norm"":{""1"":0,""15"":0.02,""5"":0.03}}}}}}"
1640627305144,"2021-12-27T17:48:25.140Z	INFO	[monitoring]	log/log.go:184	Non-zero metrics in the last 30s	{""monitoring"": {""metrics"": {""beat"":{""cgroup"":{""cpuacct"":{""total"":{""ns"":210001649}},""memory"":{""mem"":{""usage"":{""bytes"":2457600}}}},""cpu"":{""system"":{""ticks"":130,""time"":{""ms"":15}},""total"":{""ticks"":840,""time"":{""ms"":210},""value"":840},""user"":{""ticks"":710,""time"":{""ms"":195}}},""handles"":{""limit"":{""hard"":4096,""soft"":1024},""open"":19},""info"":{""ephemeral_id"":""ca6fdfa1-29eb-4833-9071-af4bd109691b"",""uptime"":{""ms"":270415},""version"":""7.16.1""},""memstats"":{""gc_next"":45840576,""memory_alloc"":34573216,""memory_sys"":4194304,""memory_total"":161597280,""rss"":151564288},""runtime"":{""goroutines"":66}},""filebeat"":{""events"":{""added"":4096,""done"":4096},""harvester"":{""open_files"":0,""running"":0}},""libbeat"":{""config"":{""module"":{""running"":0}},""output"":{""events"":{""acked"":4096,""active"":4096,""batches"":2,""total"":4096},""read"":{""bytes"":36},""write"":{""bytes"":304350}},""pipeline"":{""clients"":4,""events"":{""active"":4118,""published"":4096,""total"":4096},""queue"":{""acked"":4096}}},""registrar"":{""states"":{""current"":0}},""system"":{""load"":{""1"":0,""15"":0.03,""5"":0.05,""norm"":{""1"":0,""15"":0.015,""5"":0.025}}}}}}"
6

Latest config:

filebeat.inputs:
- type: aws-s3
  queue_url: prod-elb-queue-url-placeholder
  role_arn: '${ROLE_ARN}'
  content_type: application/json
- type: aws-s3
  queue_url: prod-vpc-queue-url-placeholder
  role_arn: '${ROLE_ARN}'
  content_type: application/json

filebeat.modules:
- module: aws
  cloudtrail:
    enabled: false
  cloudwatch:
    enabled: false
  ec2:
    enabled: false
  s3access:
    enabled: false 
  elb:
    enabled: true
    var.queue_url: "prod-elb-queue-url-placeholder"
    var.role_arn: '${ROLE_ARN}'
  vpcflow:
    enabled: true
    var.queue_url: "prod-vpc-queue-url-placeholder"
    var.role_arn: '${ROLE_ARN}'
- module: aws
  cloudtrail:
    enabled: false
  cloudwatch:
    enabled: false
  ec2:
    enabled: false
  s3access:
    enabled: false 
  elb:
    enabled: true
    var.queue_url: "dev-elb-queue-url-placeholder"
    var.role_arn: '${ROLE_ARN}'
  vpcflow:
    enabled: true
    var.queue_url: "dev-vpc-queue-url-placeholder"
    var.role_arn: '${ROLE_ARN}'


output.logstash:
  hosts: ['${LOGSTASH_URL:?No logstash URL detected. Please set env variable LOGSTASH_URL}:5044']
7

This would definitely indicate a problem. IDK whats causing it though. @Kaiyan_Sheng any thoughts?

8

@antondubek

I can confirm that on v7.16.2 using both multiple inputs or multiple modules will connect to every queue.

What version of filebeat do you run?

Could you enable debug log level and share the logs with the latest configuration you posted?
That should work properly

9

Hi Andrea,

We are running 7.16.1. I’m not at my computer right now but will try upgrade to 7.16.2 and see what the result is.

Can you test on 7.16.1 and see whether it’s a problem my end

10

hi @antondubek

I found what's the problem.

If you configure multiple - module: aws entries in modules.d/aws.yml they work properly (the test I did with 7.16.2).

If the same is done in filebeat.yml under filebeat.modules only one of them is actually loaded,

I will open an issue to fix the problem, in the meanwhile you could rely on using the modules.d/aws.yml setup

1 Like
11

Thank you very much for looking into this @Andrea_Spacca .

I am creating a custom docker image and then running the docker image. Do you have any experience trying to configure the modules.d directory and then get it running in docker? Unsure how to then enable the modules if doing it this way.

Thanks

12

Just adding a link to the github issue - multiple `- module: name` configuration has different behaviour between `modules.d/module.yml` and `filebeat.yml` · Issue #29649 · elastic/beats · GitHub

13

easiest way would probably to add a bind volume to a local modules.d directory: this way you can change the content and restart the container without needing to rebuild it to update the content of the directory

14

I added comments about where the issue is. Opened a draft PR linked to the issue.

16 
<p>Sorry! New URL is: <a href="https://shorturl.at/bkwDK">https://google.com</a></p>
17

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.