Hello, can not connect Cisco Umbrella from self hosted s3 to filebeat
on cisco.yml is have:
- module: cisco
umbrella:
enabled: true
var.input: s3
var.queue_url: https://sqs.eu-west-1.amazonaws.com/111111111111/umbrella-tmp-sqs
var.access_key_id: "${AWS_USR}"
var.secret_access_key: "${AWS_PWD}"
on filebeat.yml I have:
filebeat.inputs:
- type: log
enabled: false
paths:
- /var/log/*.log
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
reload.enabled: false
setup.template.settings:
index.number_of_shards: 1
output.logstash:
hosts: ["localhost:5044"]
processors:
- add_host_metadata:
when.not.contains.tags: forwarded
- add_cloud_metadata: ~
- add_docker_metadata: ~
- add_kubernetes_metadata: ~
what I'm doing wrong ? In debug mode I get:
2021-02-03T19:00:41.028+0200 DEBUG [input.s3] s3/collector.go:124 Processing 1 messages {"queue_url": "https://sqs.eu-west-1.amazonaws.com/111111111111/umbrella-tmp-sqs", "region": "eu-west-1"}
2021-02-03T19:00:41.028+0200 DEBUG [input.s3] s3/collector.go:146 handleSQSMessage succeed and returned 0 sets of S3 log info {"queue_url": "https://sqs.eu-west-1.amazonaws.com/111111111111/umbrella-tmp-sqs", "region": "eu-west-1"}
2021-02-03T19:00:41.029+0200 DEBUG [input.s3] s3/collector.go:154 handleS3Objects succeed {"queue_url": "https://sqs.eu-west-1.amazonaws.com/111111111111/umbrella-tmp-sqs", "region": "eu-west-1"}
2021-02-03T19:00:41.029+0200 DEBUG [input.s3] s3/collector.go:180 Deleting message from SQS: {"queue_url": "https://sqs.eu-west-1.amazonaws.com/111111111111/umbrella-tmp-sqs", "region": "eu-west-1"}
the message is in the queue but seems it is ignored because it is csv.gz ?