# S3 Output Plugin: Correct Way to manage codec

**URL:** <https://discuss.elastic.co/t/s3-output-plugin-correct-way-to-manage-codec/282103>\
**Category:** Logstash\
**Created:** [August 20, 2021, 5:12pm UTC](https://discuss.elastic.co/t/s3-output-plugin-correct-way-to-manage-codec/282103 "2021-08-20T17:12:43Z")\
**Posts on this page:** 3\
**Page:** 1

<div class="post-metadata">

**Author:** ![Hammond95](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hammond95/32/93591_2.png) [@Hammond95](https://discuss.elastic.co/u/Hammond95)\
**Post date:** [August 20, 2021, 5:12pm UTC](https://discuss.elastic.co/t/s3-output-plugin-correct-way-to-manage-codec/282103/1 "2021-08-20T17:12:43Z")

</div>

Hi everyone, I have the following problem:  
We have in place a pipeline which consist of:

```auto
[PrestoDB Clusters] ==auditing==> [Kafka] <== [Logstash] ==> Elastic + S3

```

The auditing messages on kafka are basically json messages composed of various fields which may contain **ANY character typable by the user**.

The ingestion on elastic works almost with no problems.  
Now I wanted to write on s3 some fields (but potentially all of them) in a text file or eventually in a parquet file.

So I am using the S3 Output Plugin,  
I had to configure it in this way to make it somehow work, but obviously I am facing many problem due to characters like newlines, delimiters, strange characters etc… Also this doesn’t seem like a good approach since I have 20-30 more fields to put.

```auto
s3 {
        region => "eu-west-1"
        bucket => "my-bucket"
        prefix => "audit/some/sub/folder"
        encoding => "none"
        rotation_strategy => "size_and_time"
        temporary_directory => "/tmp/logstash"
        upload_queue_size => 4
        upload_workers_count => 4
        size_file => 5242880
        time_file => 2
        codec => line {
          format => "%{[CreateDate]}|%{[orgId]}|%{[QueryID]}|%{[Catalog]}|%{[User]}|%{[Query]}|%{[QueryStartTime]}|%{[EventName]}|%{[QueryType]}|%{[QueryEndTime]}"
}

```

I have also tried the json codec, which does the job pretty well but I don’t want to write data in json format, since the files will be read in Presto/spark clusters by data scientists and it is not convenient to parse json with these tools.

I have tried with the csv codec but it doesn’t work at all, and I couldn’t understand why…

Is there something I am missing?

---

<div class="post-metadata">

**Author:** ![Hammond95](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/hammond95/32/93591_2.png) [@Hammond95](https://discuss.elastic.co/u/Hammond95)\
**Post date:** [August 23, 2021, 5:21pm UTC](https://discuss.elastic.co/t/s3-output-plugin-correct-way-to-manage-codec/282103/2 "2021-08-23T17:21:01Z")

</div>

I managed to solve my problem.

I missed it both from the logs and the docs, but actually this codec plugin (csv) doesn't come installed, so you have to install it first with:

`bin/logstash-plugin install logstash-codec-csv`

After that I had to escape newline from fields to avoid unwanted line breaks:

```auto
mutate {
    gsub => ["[Query]", "[\n]", "\\\\n" ]
    gsub => ["[PreparedQuery]", "[\n]", "\\\\n" ]
}

```

The plugin will take care of doubling any double quote in the data (that's how you escape doublequotes).

For the separator I have opened another thread in the forum, see:

> [@S3 CSV Codec - Using a non printable character as separator](https://discuss.elastic.co/t/s3-csv-codec-using-a-non-printable-character-as-separator/282262):
>
> Is it possible to use a non printable character as a separator in the csv codec? codec =\> csv { columns =\> ["col1", "col2", "col3"] charset =\> "UTF-8" separator =\> "\\u001F" } I have tried the following: \u001f, \u001F, \\u001f, \\u001F, \u{001F} some ideas? If this is not possible, how would I have to modify the code to make it possible?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [September 20, 2021, 5:21pm UTC](https://discuss.elastic.co/t/s3-output-plugin-correct-way-to-manage-codec/282103/3 "2021-09-20T17:21:08Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.