# Sending data to multiple outputs with different parsing , we are using s3 input plugin

**URL:** https://discuss.elastic.co/t/sending-data-to-multiple-outputs-with-different-parsing-we-are-using-s3-input-plugin/354566
**Category:** Logstash
**Created:** [March 2, 2024, 12:24pm UTC](https://discuss.elastic.co/t/sending-data-to-multiple-outputs-with-different-parsing-we-are-using-s3-input-plugin/354566 "2024-03-02T12:24:06Z")
**Posts on this page:** 3
**Page:** 1

<div class="post-metadata">

### Author: ![Sumit\_G](https://avatars.discourse-cdn.com/v4/letter/s/4da419/32.png) [@Sumit\_G](https://discuss.elastic.co/u/Sumit_G)
#### Post date: [March 2, 2024, 12:24pm UTC](https://discuss.elastic.co/t/sending-data-to-multiple-outputs-with-different-parsing-we-are-using-s3-input-plugin/354566/1 "2024-03-02T12:24:06Z")

</div>

This is the setup that we need:  
Input Plugin - S3

Output plugin: Elastic Search and S3

However, the requirement is that the data that will be going to Elasticsearch won't be the same as that of the data going to S3.

The data going to s3 will need more mutation in comparison to the data going to Elasticsearch.

I don't see that we can use a filter(for mutating) in the output option of logstash.

Can you please help on how this can be achieved?

---

<div class="post-metadata">

### Author: ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)
#### Post date: [March 2, 2024, 1:08pm UTC](https://discuss.elastic.co/t/sending-data-to-multiple-outputs-with-different-parsing-we-are-using-s3-input-plugin/354566/2 "2024-03-02T13:08:51Z")

</div>

> [@Sumit\_G](#):
>
> I don't see that we can use a filter(for mutating) in the output option of logstash.

It is not possible to use filters in the output, every transformation in the data needs to be done in the `filter` block.

What you need is to use multiple pipelines, check the documentation for the [forked-path pattern](https://www.elastic.co/guide/en/logstash/current/pipeline-to-pipeline.html#forked-path-pattern).

You will need to have 3 pipelines, one will have your input and common filters and will output to the other two pipelines, one of them will output to elasticsearch, and the other will have the extra mutations and output to s3.

You need something like this:

`pipelines.yml`

```auto
- pipeline.id: input-s3
  path.config: /path/to/input-s3.conf
- pipeline.id: output-es
  path.config: /path/to/output-es.conf
- pipeline.id: output-s3
  path.config: /path/to/output-s3.conf

```

`input-s3.conf`

```auto
input {
  s3 { s3 input configuration}
}
filter {
  common filter configuration
}
output {
  pipeline {
    send_to => ["output-es","output-s3"]
  }
}

```

`output-es.conf`

```auto
input {
  pipeline {
    address => "output-es"
  }
}
filter {
  filters for es output only
}
output {
  elasticsearch {
    your es output
  }
}

```

`output-s3.conf`

```auto
input {
  pipeline {
    address => "output-s3"
  }
}
filter {
  filters for s3 output only
}
output {
  s3 {
    your s3 output
  }
}

```

---

<div class="post-metadata">

### Author: ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)
#### Post date: [March 30, 2024, 1:09pm UTC](https://discuss.elastic.co/t/sending-data-to-multiple-outputs-with-different-parsing-we-are-using-s3-input-plugin/354566/3 "2024-03-30T13:09:21Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
