Passs Json input from s3 files to elasticsearch

Elastic Stack Logstash
1

I am trying to pass my data from s3 to elasticsearch 1
THE FILES ARE IN S3.
this is my config file

input {
s3 {
bucket => "awsesbucket1rec"
access_key_id => "AKIAVU3QDW2HVZ2NFV6I"
secret_access_key => "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"
region => "us-east-1"
codec => "json"
}
}

filter{
mutate{
add_field =>{
"file" => "%{[@metadata][s3][key]}"
}
}

mutate {
gsub => [ "file", ".\w{3}", "", "file", "([0-9]{4}-[0-9]{2}-[0-9]{2})-[0-9]+", "\1" ] }
}

output {
amazon_es {
hosts => ["https://vpc-log-service-legacy-data-lm4aq2wmxzh4xxj2uukkucykfi.us-east-1.es.amazonaws.com"]
region => "us-east-1"
aws_access_key_id => 'AKIAVU3QDW2HVZ2NFV6I'
aws_secret_access_key => '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'
index => "%{file}"
template_name => "sqe_template"
template_overwrite => "true"
codec => "json"
}
}

and this is my data

[{"SessionID":"304257638164423-16b15e39fdd-92a7fa40-3590-471f-8b3b-a1d34438e5c9","Study":"eCOA261_2019JAN04","Locale":"en_US","ProdVer":"2.0.0","StudyVer":"1.0.0","Sponsor":"ERT","Device":"3535","SubjectID":"no subject data","Type":"SessionStart","Level":"INFO","Message":"Session","Source":"Web App","Location":"actions.transmitLogs","Site":"0009","Environment":"2.6.1 (pht-syn-study06 : eCOA261_2019JAN04)","EnvironmentID":"staging","Role":"site","KRPT":"no subject data","EventOrder":"17","UserID":"442","SigID":"undefined","Protocol":"TestStudy","ClientTime":"2019-06-02T01:52:47.580"},{"SessionID":"304257638164423-16b15e39fdd-8b4a517d-769a-4376-8b23-e5a5b61e0efb","Study":"eCOA261_2019JAN04","Locale":"en_US","ProdVer":"2.0.0","StudyVer":"1.0.0","Sponsor":"ERT","Device":"3535","SubjectID":"no subject data","Type":"SessionStart","Level":"INFO","Message":"Session","Source":"Web App","Location":"actions.transmitLogs","Site":"0009","Environment":"2.6.1 (pht-syn-study06 : eCOA261_2019JAN04)","EnvironmentID":"staging","Role":"site","KRPT":"no subject data","EventOrder":"16","UserID":"442","SigID":"undefined","Protocol":"TestStudy","ClientTime":"2019-06-02T01:52:47.580"}]

I get errors if I pass it in this format, do I change the format or change the configuration file for logstash?

2

I think you should share more details about the exact error messages you see, where they are, etc.
I'm guessing it's in the logstash logs but why didn't you share those here, properly formatted, or via a gist? Saying you have an error and not including it, is a bit weird.

3

I am sorry! Ill be sharing the error I encounter!

4

[ERROR] 2019-06-07 14:22:46.447 [[main]<s3] json - JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: Unexpected character ('.' (code 46)): expected a valid value (number, String, array, object, 'true', 'false' or 'null')

[ERROR] 2019-06-07 14:22:46.566 [[main]<s3] json - JSON parse error, original data now in message field {:error=>#<LogStash::Json::ParserError: Unrecognized token 'End': was expecting ('true', 'false' or 'null')

These are the errors I get.

Just the first record of the file goes in others are rejected and throw errors!

Any suggestion how to change the config file or do I have have my input records in this way?

{data1},
{data2},
{data3}

Kindly give your suggestions!1

Thank You,
Mohit Ruke

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.