Hello,
We have noticed that functionbeat is pushing way to many logs to cloudwatch, and is resulting in our bill being high, as we are getting charged for PutLogEvents. I have set the logging level to be error, but can still see INFO and DEBUG in the functionbeat cloudwatch log stream.
- level: error
metrics:
- enabled: true
period: 60
15:52:39
2019-11-01T15:52:39.177Z DEBUG [processors] processors/processor.go:66 Processors: decode_json_fields=message, drop_fields=message_type, subscription_filters, beat.version, beat.hostname, _type, _score, log_stream, _index
15:52:39
2019-11-01T15:52:39.182Z INFO elasticsearch/client.go:163 Elasticsearch url: http://10.10.110.48:9200
15:52:39
2019-11-01T15:52:39.184Z DEBUG [publish] pipeline/consumer.go:137 start pipeline event consumer
15:52:39
2019-11-01T15:52:39.184Z INFO [publisher] pipeline/module.go:110 Beat name: ip-10-130-48-49
15:52:39
2019-11-01T15:52:39.184Z INFO [monitoring] log/log.go:117 Starting metrics logging every 30s
15:52:39
2019-11-01T15:52:39.185Z INFO instance/beat.go:400 functionbeat start running.
15:52:39
2019-11-01T15:52:39.185Z INFO [functionbeat] beater/functionbeat.go:74 Functionbeat is running
15:52:39
2019-11-01T15:52:39.186Z INFO elasticsearch/client.go:163 Elasticsearch url: http://10.10.110.48:9200
15:52:39
2019-11-01T15:52:39.186Z INFO [license-manager] licenser/manager.go:184 License manager started, retrieving initial license
15:52:39
2019-11-01T15:52:39.186Z INFO [functionbeat] licenser/manager.go:331 Waiting on synchronous license check
15:52:39
2019-11-01T15:52:39.186Z DEBUG [license-manager] licenser/manager.go:230 Starting periodic license check, refresh: 15m0s grace: 45m0s
15:52:39
2019-11-01T15:52:39.576Z DEBUG [elasticsearch] elasticsearch/client.go:689 ES Ping(url=http://10.10.110.48:9200)
15:52:39
2019-11-01T15:52:39.623Z DEBUG [elasticsearch] elasticsearch/client.go:712 Ping status code: 200
15:52:39
2019-11-01T15:52:39.623Z INFO elasticsearch/client.go:713 Connected to Elasticsearch version 6.3.2
Also, it appears it is outputting the messages of the cloudwatch log we are pulling from to push to ES, which is resulting in the message being sent to CW twice.
This is seen via the publish event -
2019-11-01T15:52:39.674Z DEBUG [publish] pipeline/processor.go:308 Publish event: {
Any help would be much appreciated. FYI - We are running 6.4.0 version of functionbeat.
