Logstash s3 plugin to push json files to elasticsearch

Elastic Stack Logstash
1

Usecase

Push s3 JSON objects (s3 containers json files), each object will be one document)

I have written a pipeline

input {
    s3 {
        access_key_id => "MY_KEY"
        secret_access_key => "MY_SECRET"
        bucket => "sthreetoes"
        region => "ap-south-1"
        codec => "json"
    }
}
output {
    elasticsearch {
        hosts => "http://elasticsearch:9200"
        index => "candidate-data"
    }
}

When I run the pipeline, losgstash is shipping to index considering each key-value paid as document.

json object in s3

{
    "id": 246,
    "first_name": "Lell",
    "last_name": "Bsel",
    "email": "lbros6t@sakura.jp",
    "gender": "Female",
    "ip_address": "12.12.12.8"
}

When we run the logstash pipeline index is created with 6 documents considering each key-value pair in JSON as a message.

Any solution to save the JSON object as a single document

2

It sounds like you need a multiline codec on the input.

3

Yes, we are going with one line will not help pickup the whole json.