# Configure Filebeat to not delete AWS SQS message if the message does not match the file\_selectors

**URL:** https://discuss.elastic.co/t/configure-filebeat-to-not-delete-aws-sqs-message-if-the-message-does-not-match-the-file-selectors/325299
**Category:** Beats
**Tags:** filebeat
**Created:** [February 10, 2023, 9:21pm UTC](https://discuss.elastic.co/t/configure-filebeat-to-not-delete-aws-sqs-message-if-the-message-does-not-match-the-file-selectors/325299 "2023-02-10T21:21:11Z")
**Posts on this page:** 3
**Page:** 1

<div class="post-metadata">

### Author: ![b2ron](https://avatars.discourse-cdn.com/v4/letter/b/e495f1/32.png) [@b2ron](https://discuss.elastic.co/u/b2ron)
#### Post date: [February 10, 2023, 9:21pm UTC](https://discuss.elastic.co/t/configure-filebeat-to-not-delete-aws-sqs-message-if-the-message-does-not-match-the-file-selectors/325299/1 "2023-02-10T21:21:11Z")

</div>

AWS S3 is configured to send event notification to SQS queue

Filebeat is using aws-s3 to pull logs from S3 through the SQS queue

```auto
filebeat.inputs:
  - type: aws-s3
    queue_url: https://sqs.us-east-2.amazonaws.com/aws-id/queue-name
    gzip: true
    file_selectors:
      - regex: 'log/app1'

```

Filebeat deletes all the messages present in the queue, whereas it should only process and delete the messages matching the `file_selectors` regex

Question: How to configure filebeat to **not** delete SQS queue message if the message does not match the `file_selectors` regex?

P.S.

The use case that I am trying to achieve is

1. AWS S3 notification -\> SQS queue(all type of files)
2. Multiple filebeat instances configured with the same queue

- Filebeat-1: configured `file_selectors` for log/app1. filebeat output: Logstash
- Filebeat-2: configured `file_selectors` for log/app2. filebeat output: Elasticsearch, index app2  
or
- Filebeat-1: configured `file_selectors` for log/app1. filebeat output: Elasticsearch, index app1
- Filebeat-2: configured `file_selectors` for log/app2. filebeat output: Elasticsearch, index app2

---

<div class="post-metadata">

### Author: ![Ayush\_Mathur](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ayush_mathur/32/77134_2.png) [@Ayush\_Mathur](https://discuss.elastic.co/u/Ayush_Mathur)
#### Post date: [February 13, 2023, 7:29am UTC](https://discuss.elastic.co/t/configure-filebeat-to-not-delete-aws-sqs-message-if-the-message-does-not-match-the-file-selectors/325299/2 "2023-02-13T07:29:37Z")

</div>

Hello @b2ron , the whole idea of queue is to delete the message from the queue as soon as one of the subscriber reads it from the queue. What you are trying to achieve can be done using a single beat instance rather than running multiple beat instances for each `file_selector`.  
You can either add conditional processors to set index names or configure different index names depending on (meta)data of the event in output section and push the events to required hop (logstash or Elasticsearch).

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [March 13, 2023, 9:30am UTC](https://discuss.elastic.co/t/configure-filebeat-to-not-delete-aws-sqs-message-if-the-message-does-not-match-the-file-selectors/325299/3 "2023-03-13T09:30:05Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.