Configure Filebeat to not delete AWS SQS message if the message does not match the file_selectors

[b2ron]

AWS S3 is configured to send event notification to SQS queue

Filebeat is using aws-s3 to pull logs from S3 through the SQS queue

filebeat.inputs:
  - type: aws-s3
    queue_url: https://sqs.us-east-2.amazonaws.com/aws-id/queue-name
    gzip: true
    file_selectors:
      - regex: 'log/app1'

Filebeat deletes all the messages present in the queue, whereas it should only process and delete the messages matching the file_selectors regex

Question: How to configure filebeat to not delete SQS queue message if the message does not match the file_selectors regex?

P.S.

The use case that I am trying to achieve is

1. AWS S3 notification -> SQS queue(all type of files)
2. Multiple filebeat instances configured with the same queue

• Filebeat-1: configured file_selectors for log/app1. filebeat output: Logstash
• Filebeat-2: configured file_selectors for log/app2. filebeat output: Elasticsearch, index app2
  or
• Filebeat-1: configured file_selectors for log/app1. filebeat output: Elasticsearch, index app1
• Filebeat-2: configured file_selectors for log/app2. filebeat output: Elasticsearch, index app2

[Ayush_Mathur]

Hello @b2ron , the whole idea of queue is to delete the message from the queue as soon as one of the subscriber reads it from the queue. What you are trying to achieve can be done using a single beat instance rather than running multiple beat instances for each file_selector.
You can either add conditional processors to set index names or configure different index names depending on (meta)data of the event in output section and push the events to required hop (logstash or Elasticsearch).