S3 output plugin generates invalid json

Elastic Stack Logstash
1

Hi,

We are using s3 output plugin to forward logs into an s3 bucket with json_lines codec. The files generated are raw text files with each line containing a single quoted json file with special chars such as \n in it.

    output{
      s3{
       access_key_id => "****"
       secret_access_key => "****"
       region => "****"
       bucket => "****"
       time_file => 5
       rotation_strategy => "time"
       codec => "json_lines"
     }
    }

These lines couldnt be processed for example by a tool like jq or even python json libraries.

Is there any way to reprocess these files already generated? as we have GBs of such logs generated.

And is there a way to avoid this single quoted jsons with special chars to be output by the plugin?

Thank you

2

Looks these files we received were post processed outside of logstash using a python library. Causing this issue.

Updating the cause as I have identified the resolution.

3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.