# Cisco Umbrella selfhosted s3 the queue is not processing **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [February 3, 2021, 5:09pm UTC](https://discuss.elastic.co/t/cisco-umbrella-selfhosted-s3-the-queue-is-not-processing/263152 "2021-02-03T17:09:11Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![YegorKovylyayev](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yegorkovylyayev/32/83323_2.png) [@YegorKovylyayev](https://discuss.elastic.co/u/YegorKovylyayev)\ **Post date:** [February 3, 2021, 5:09pm UTC](https://discuss.elastic.co/t/cisco-umbrella-selfhosted-s3-the-queue-is-not-processing/263152/1 "2021-02-03T17:09:11Z")
Hello, can not connect Cisco Umbrella from self hosted s3 to filebeat on cisco.yml is have: ```auto - module: cisco umbrella: enabled: true var.input: s3 var.queue_url: https://sqs.eu-west-1.amazonaws.com/111111111111/umbrella-tmp-sqs var.access_key_id: "${AWS_USR}" var.secret_access_key: "${AWS_PWD}" ``` on filebeat.yml I have: filebeat.inputs: ```auto - type: log enabled: false paths: - /var/log/*.log filebeat.config.modules: path: ${path.config}/modules.d/*.yml reload.enabled: false setup.template.settings: index.number_of_shards: 1 output.logstash: hosts: ["localhost:5044"] processors: - add_host_metadata: when.not.contains.tags: forwarded - add_cloud_metadata: ~ - add_docker_metadata: ~ - add_kubernetes_metadata: ~ ``` what I'm doing wrong ? In debug mode I get: ```auto 2021-02-03T19:00:41.028+0200 DEBUG [input.s3] s3/collector.go:124 Processing 1 messages {"queue_url": "https://sqs.eu-west-1.amazonaws.com/111111111111/umbrella-tmp-sqs", "region": "eu-west-1"} 2021-02-03T19:00:41.028+0200 DEBUG [input.s3] s3/collector.go:146 handleSQSMessage succeed and returned 0 sets of S3 log info {"queue_url": "https://sqs.eu-west-1.amazonaws.com/111111111111/umbrella-tmp-sqs", "region": "eu-west-1"} 2021-02-03T19:00:41.029+0200 DEBUG [input.s3] s3/collector.go:154 handleS3Objects succeed {"queue_url": "https://sqs.eu-west-1.amazonaws.com/111111111111/umbrella-tmp-sqs", "region": "eu-west-1"} 2021-02-03T19:00:41.029+0200 DEBUG [input.s3] s3/collector.go:180 Deleting message from SQS: {"queue_url": "https://sqs.eu-west-1.amazonaws.com/111111111111/umbrella-tmp-sqs", "region": "eu-west-1"} ``` the message is in the queue but seems it is ignored because it is csv.gz ? ---
**Author:** ![Marius\_Iversen](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/marius_iversen/32/68988_2.png) [@Marius\_Iversen](https://discuss.elastic.co/u/Marius_Iversen)\ **Post date:** [February 9, 2021, 2:12pm UTC](https://discuss.elastic.co/t/cisco-umbrella-selfhosted-s3-the-queue-is-not-processing/263152/2 "2021-02-09T14:12:29Z")
Hello @YegorKovylyayev . I don't think there should be any reason why the messages should be ignored, unless the account related to the access\_key and secret does not have access to read the logs. It might be other debug logs that could be useful compared to only the 4 lines added to the post, would you mind deleting/moving the existing logfile, starting it up again, and let it run for a few minutes and share the output? ---
**Author:** ![YegorKovylyayev](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yegorkovylyayev/32/83323_2.png) [@YegorKovylyayev](https://discuss.elastic.co/u/YegorKovylyayev)\ **Post date:** [February 9, 2021, 2:58pm UTC](https://discuss.elastic.co/t/cisco-umbrella-selfhosted-s3-the-queue-is-not-processing/263152/3 "2021-02-09T14:58:59Z")
Hello @Marius_Iversen I've created s3, sns, sqs from the same account and I can get logs from aws cli from the same account here is log [Log](https://pastebin.com/raw/tQRmcjEV) ---
**Author:** ![Kaiyan\_Sheng](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kaiyan_sheng/32/38247_2.png) [@Kaiyan\_Sheng](https://discuss.elastic.co/u/Kaiyan_Sheng)\ **Post date:** [February 9, 2021, 5:06pm UTC](https://discuss.elastic.co/t/cisco-umbrella-selfhosted-s3-the-queue-is-not-processing/263152/4 "2021-02-09T17:06:31Z")
Hi @YegorKovylyayev thanks for the log! Looks like Filebeat is able to see the SQS message but not able to see which S3 bucket/log it is pointing to. Do you have the SQS-S3 setup so when a new log entry gets into S3, a SQS message will be created? If the SQS-S3 setup is good, then is it possible to send us the actual SQS message content? Also what version of Filebeat are you running? If you could upgrade to the latest version, it will show more debug level message I believe. Thanks! ---
**Author:** ![YegorKovylyayev](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yegorkovylyayev/32/83323_2.png) [@YegorKovylyayev](https://discuss.elastic.co/u/YegorKovylyayev)\ **Post date:** [February 9, 2021, 5:46pm UTC](https://discuss.elastic.co/t/cisco-umbrella-selfhosted-s3-the-queue-is-not-processing/263152/5 "2021-02-09T17:46:07Z")
> [@Kaiyan\_Sheng](#): > > If the SQS-S3 setup is good, then is it possible to send us the actual SQS message content? > Also what version of Filebeat are you running? If you could upgrade to the latest version, i I have version 7.10.2 sure here is content and if needed I can even reproduce whole setup command line set [sqs message](https://pastebin.com/raw/sgq3qY5q) ---
**Author:** ![Kaiyan\_Sheng](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kaiyan_sheng/32/38247_2.png) [@Kaiyan\_Sheng](https://discuss.elastic.co/u/Kaiyan_Sheng)\ **Post date:** [February 10, 2021, 4:31pm UTC](https://discuss.elastic.co/t/cisco-umbrella-selfhosted-s3-the-queue-is-not-processing/263152/6 "2021-02-10T16:31:04Z")
Hmm this is odd! Thank you for the SQS message sample! Filebeat should be able to collect the s3 info from this. Could you upgrade to 7.11.0 filebeat and get some debug level logs please? I tested locally with your SQS message and it was fine to determine what's the S3 info from the message. Thanks!! ---
**Author:** ![YegorKovylyayev](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yegorkovylyayev/32/83323_2.png) [@YegorKovylyayev](https://discuss.elastic.co/u/YegorKovylyayev)\ **Post date:** [February 10, 2021, 5:33pm UTC](https://discuss.elastic.co/t/cisco-umbrella-selfhosted-s3-the-queue-is-not-processing/263152/7 "2021-02-10T17:33:46Z")
upgraded and here is full debug log after: [Logs after upgrade](https://pastebin.com/raw/UXb8uCV6) maybe I doing something wrong during configuration steps ? here is how I setup AWS part [s3 sqs sns setup](https://pastebin.com/raw/FkTHX6fB) thank you for your help ---
**Author:** ![YegorKovylyayev](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yegorkovylyayev/32/83323_2.png) [@YegorKovylyayev](https://discuss.elastic.co/u/YegorKovylyayev)\ **Post date:** [February 10, 2021, 5:43pm UTC](https://discuss.elastic.co/t/cisco-umbrella-selfhosted-s3-the-queue-is-not-processing/263152/8 "2021-02-10T17:43:27Z")
I even remove filebeat, install from scratch version 7.11 and it is doesen't help ---
**Author:** ![Kaiyan\_Sheng](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/kaiyan_sheng/32/38247_2.png) [@Kaiyan\_Sheng](https://discuss.elastic.co/u/Kaiyan_Sheng)\ **Post date:** [February 10, 2021, 6:52pm UTC](https://discuss.elastic.co/t/cisco-umbrella-selfhosted-s3-the-queue-is-not-processing/263152/9 "2021-02-10T18:52:46Z")
Ahh thanks for sharing the steps you did to setup s3-sqs. I think problem might be there. We are not using SNS at all. Connection should be between S3 and SQS directly. [Getting AWS logs from S3 using Filebeat and the Elastic Stack | Elastic Blog](https://www.elastic.co/blog/getting-aws-logs-from-s3-using-filebeat-and-the-elastic-stack) shows step by step how to setup. Hopefully this will help!! 🤞 🤞 ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [March 10, 2021, 8:52pm UTC](https://discuss.elastic.co/t/cisco-umbrella-selfhosted-s3-the-queue-is-not-processing/263152/10 "2021-03-10T20:52:48Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.