How to validate data uploaded on Elastic cluster successfully

Hi ,

I have some raw data and after preprocessing it through python I am uploading it on ELK through logstash in the following way.


/home/abc/script/ | /usr/share/logstash/bin/logstash -f /home/abc/script/budget.conf --path.settings=/etc/logstash

When I run this script ( and if it fails due to issue with python script (/home/abc/script/ It show me an error on terminal .

Now I want the check if there is any way (plugin) to display an error on terminal if it would not able to upload the data on ELK due to some error . Although I can see the error on logstash log file but that is not helpful for me as I trigger this script at some specific action and want to generate an alert email / msg if data not uploaded on ELK successfully.
I tried stdout plugin but it showing same msg on both pass/fail condition.

For testing purpose I passed the date in yyyy-m-d format and it didn't upload the data on ELK (as expected as in Elastic date format should be yyyy-mm-dd format). It show an error on logstash log file:

error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [startDate]", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"Invalid format: "2019-1-31" is malformed at

But of terminal in both success and failure cases it showing the same msg .

2019-10-17T05:41:04.125Z dev02 %{message}
2019-10-17T05:41:04.133Z dev02 %{message}

Got the answer. I can make a elastic query and check the timestamp of the index.

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.