# S3 output plugin

**URL:** <https://discuss.elastic.co/t/s3-output-plugin/297675>\
**Category:** Logstash\
**Created:** [February 19, 2022, 10:57pm UTC](https://discuss.elastic.co/t/s3-output-plugin/297675 "2022-02-19T22:57:39Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![brunoof1](https://avatars.discourse-cdn.com/v4/letter/b/c0e974/32.png) [@brunoof1](https://discuss.elastic.co/u/brunoof1)\
**Post date:** [February 19, 2022, 10:57pm UTC](https://discuss.elastic.co/t/s3-output-plugin/297675/1 "2022-02-19T22:57:39Z")

</div>

Hello everyone, everything good ?

I have 2 doubts. What are all the file formats supported by the s3 output plugin?

I'm using Oracle, kafka and file input and writing to s3 with s3 output.

I would like to create Parquet or ORC files.

Thanks

---

<div class="post-metadata">

**Author:** ![yaauie](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/yaauie/32/23363_2.png) [@yaauie](https://discuss.elastic.co/u/yaauie)\
**Post date:** [February 20, 2022, 4:15am UTC](https://discuss.elastic.co/t/s3-output-plugin/297675/2 "2022-02-20T04:15:25Z")

</div>

The s3 output saves files using whatever bytestream output of the codec it is configured with. The output receives batches of events, hands each event to the codec, and concatenates the result onto a local file based on the filename pattern it is configured with. Those local files are flushed at (configurable) intervals to S3.

The S3 output's _default_ codec is the [line codec](https://www.elastic.co/guide/en/logstash/current/plugins-outputs-s3.html#plugins-outputs-s3-codec), which stringifies the event, and results in one line of content in the file per event. If you configure the s3 output with the json lines codec, it outputs newline-delimited JSON. And if you configure it with the syslog codec, it outputs lines that are syslog-encoded, etc.

---

<div class="post-metadata">

**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\
**Post date:** [February 20, 2022, 4:38am UTC](https://discuss.elastic.co/t/s3-output-plugin/297675/3 "2022-02-20T04:38:14Z")

</div>

> [@brunoof1](#):
>
> I would like to create Parquet or ORC files.

As I understand it, both ORC and Parquet are columnar file formats. Assuming each event is a "row" of data ...

The two formats are paged/striped. I think it is conceivable that you could write a codec with a multi\_receive\_encoded method (which consumes a batch of events) and transforms them from row-wise to column-wise. But it would be a new development effort. I do not use these formats, so I have absolutely no idea whether anyone would be interested if you created a github project and started writing this.

It also depends on whether the pages/stripes contain information about oneanother. If the output can just append the codec's encoding of a batch to its output (e.g. a file) then it works. If the previous page/stripe has to tell you where the next one is then it may not work.

---

<div class="post-metadata">

**Author:** ![brunoof1](https://avatars.discourse-cdn.com/v4/letter/b/c0e974/32.png) [@brunoof1](https://discuss.elastic.co/u/brunoof1)\
**Post date:** [February 21, 2022, 12:59pm UTC](https://discuss.elastic.co/t/s3-output-plugin/297675/4 "2022-02-21T12:59:28Z")

</div>

Thank you, @yaauie and @Badger !

Do you have a documentation that describes all codecs possibilities ?

Is there any way to change the extension from txt to json?

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/7/f/7f2e70e6fd09034e8b1e27f89a481e74f942ef82.jpeg)

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 21, 2022, 12:59pm UTC](https://discuss.elastic.co/t/s3-output-plugin/297675/5 "2022-03-21T12:59:34Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
