# Passs Json input from s3 files to elasticsearch

**URL:** <https://discuss.elastic.co/t/passs-json-input-from-s3-files-to-elasticsearch/184663>\
**Category:** Logstash\
**Created:** [June 6, 2019, 8:34pm UTC](https://discuss.elastic.co/t/passs-json-input-from-s3-files-to-elasticsearch/184663 "2019-06-06T20:34:45Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![Mohit\_Ruke](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mohit_ruke/32/46323_2.png) [@Mohit\_Ruke](https://discuss.elastic.co/u/Mohit_Ruke)\
**Post date:** [June 6, 2019, 8:34pm UTC](https://discuss.elastic.co/t/passs-json-input-from-s3-files-to-elasticsearch/184663/1 "2019-06-06T20:34:45Z")

</div>

I am trying to pass my data from s3 to elasticsearch 1  
THE FILES ARE IN S3.  
this is my config file

input {  
s3 {  
bucket =\> "awsesbucket1rec"  
access\_key\_id =\> "AKIAVU3QDW2HVZ2NFV6I"  
secret\_access\_key =\> "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"  
region =\> "us-east-1"  
codec =\> "json"  
}  
}

filter{  
mutate{  
add\_field =\>{  
"file" =\> "%{[@metadata][s3][key]}"  
}  
}

mutate {  
gsub =\> ["file", ".\w{3}", "", "file", "([0-9]{4}-[0-9]{2}-[0-9]{2})-[0-9]+", "\1" ] }  
}

output {  
amazon\_es {  
hosts =\> ["[https://vpc-log-service-legacy-data-lm4aq2wmxzh4xxj2uukkucykfi.us-east-1.es.amazonaws.com](https://vpc-log-service-legacy-data-lm4aq2wmxzh4xxj2uukkucykfi.us-east-1.es.amazonaws.com)"]  
region =\> "us-east-1"  
aws\_access\_key\_id =\> 'AKIAVU3QDW2HVZ2NFV6I'  
aws\_secret\_access\_key =\> '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'  
index =\> "%{file}"  
template\_name =\> "sqe\_template"  
template\_overwrite =\> "true"  
codec =\> "json"  
}  
}

and this is my data

[{"SessionID":"304257638164423-16b15e39fdd-92a7fa40-3590-471f-8b3b-a1d34438e5c9","Study":"eCOA261\_2019JAN04","Locale":"en\_US","ProdVer":"2.0.0","StudyVer":"1.0.0","Sponsor":"ERT","Device":"3535","SubjectID":"no subject data","Type":"SessionStart","Level":"INFO","Message":"Session","Source":"Web App","Location":"actions.transmitLogs","Site":"0009","Environment":"2.6.1 (pht-syn-study06 : eCOA261\_2019JAN04)","EnvironmentID":"staging","Role":"site","KRPT":"no subject data","EventOrder":"17","UserID":"442","SigID":"undefined","Protocol":"TestStudy","ClientTime":"2019-06-02T01:52:47.580"},{"SessionID":"304257638164423-16b15e39fdd-8b4a517d-769a-4376-8b23-e5a5b61e0efb","Study":"eCOA261\_2019JAN04","Locale":"en\_US","ProdVer":"2.0.0","StudyVer":"1.0.0","Sponsor":"ERT","Device":"3535","SubjectID":"no subject data","Type":"SessionStart","Level":"INFO","Message":"Session","Source":"Web App","Location":"actions.transmitLogs","Site":"0009","Environment":"2.6.1 (pht-syn-study06 : eCOA261\_2019JAN04)","EnvironmentID":"staging","Role":"site","KRPT":"no subject data","EventOrder":"16","UserID":"442","SigID":"undefined","Protocol":"TestStudy","ClientTime":"2019-06-02T01:52:47.580"}]

I get errors if I pass it in this format, do I change the format or change the configuration file for logstash?

---

<div class="post-metadata">

**Author:** ![martinr\_ubi](https://avatars.discourse-cdn.com/v4/letter/m/b5e925/32.png) [@martinr\_ubi](https://discuss.elastic.co/u/martinr_ubi)\
**Post date:** [June 6, 2019, 10:38pm UTC](https://discuss.elastic.co/t/passs-json-input-from-s3-files-to-elasticsearch/184663/2 "2019-06-06T22:38:39Z")

</div>

> [@Mohit\_Ruke](#):
>
> I get errors

I think you should share more details about the exact error messages you see, where they are, etc.  
I'm guessing it's in the logstash logs but why didn't you share those here, properly formatted, or via a gist? Saying you have an error and not including it, is a bit weird.

---

<div class="post-metadata">

**Author:** ![Mohit\_Ruke](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mohit_ruke/32/46323_2.png) [@Mohit\_Ruke](https://discuss.elastic.co/u/Mohit_Ruke)\
**Post date:** [June 7, 2019, 2:07pm UTC](https://discuss.elastic.co/t/passs-json-input-from-s3-files-to-elasticsearch/184663/3 "2019-06-07T14:07:54Z")

</div>

I am sorry! Ill be sharing the error I encounter!

---

<div class="post-metadata">

**Author:** ![Mohit\_Ruke](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/mohit_ruke/32/46323_2.png) [@Mohit\_Ruke](https://discuss.elastic.co/u/Mohit_Ruke)\
**Post date:** [June 7, 2019, 2:28pm UTC](https://discuss.elastic.co/t/passs-json-input-from-s3-files-to-elasticsearch/184663/4 "2019-06-07T14:28:32Z")

</div>

[ERROR] 2019-06-07 14:22:46.447 [[main]\<s3] json - JSON parse error, original data now in message field {:error=\>#\<LogStash::Json::ParserError: Unexpected character ('.' (code 46)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')

[ERROR] 2019-06-07 14:22:46.566 [[main]\<s3] json - JSON parse error, original data now in message field {:error=\>#\<LogStash::Json::ParserError: Unrecognized token 'End': was expecting ('true', 'false' or 'null')

These are the errors I get.

Just the first record of the file goes in others are rejected and throw errors!

Any suggestion how to change the config file or do I have have my input records in this way?

{data1},  
{data2},  
{data3}

Kindly give your suggestions!1

Thank You,  
Mohit Ruke

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2019, 2:28pm UTC](https://discuss.elastic.co/t/passs-json-input-from-s3-files-to-elasticsearch/184663/5 "2019-07-05T14:28:35Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.