Logstash pipeline stops working

[b725 (Brian Balagot)]

I noticed today that a pipeline stopped working

Background

• events get logged into different Amazon S3 buckets paths
• a logstash pipeline is created for different bucket paths so that they run in parallel plus different worker/batch config can be given for each

All the other pipelines are running except for this one, I've

• checked the logs in /var/log/logstash/logstash-plain.log and there's no mention of this pipeline failing or anything related - for example I've seen errors in other pipelines with as BUT not for this pipeline

[WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsear
ch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"logstash-2019.11.27", :_type=>"doc", :_ro
uting=>nil}, #<LogStash::Event:0x5759be36>], :response=>{"index"=>{"_index"=>"logstash-2019.11.27", "
_type"=>"doc", "_id"=>"KRgdq24Bxd7JxF6j5Gnl", "status"=>400, "error"=>{"type"=>"mapper_parsing_except
ion", "reason"=>"failed to parse [metrics.boardTemp]", "caused_by"=>{"type"=>"illegal_argument_except
ion", "reason"=>"[float] supports only finite values, but got [NaN]"}}}}}

• I've checked the sincedb file and can see that the last date there matches the last date of the most recent log that was uploaded to elasticsearch based on a query for the output of this pipeline that stopped
• I know, because of AWS Athena, that there are updated events for this pipeline to index

It just stopped indexing and I have no information / logs to tell me what went wrong / where / when.

Please, any help would be very much appreciated.

[Mahesh_Guntumadugu]

There is parsing error while converting [metrics.boardTemp] this field .Please cross check the filters applied for this field conversion.

"reason"=>"[float] supports only finite values, but got [NaN]"

[b725 (Brian Balagot)]

@Mahesh_Guntumadugu, thanks for responding. So the error message I posted is from a different pipeline (let's call it pipeline X) - it's still running and is fine.

So my problem is that pipeline Y stopped running and there are

• no error message
• checked its sincedb file and it's stopped where the last log is indexed (which I can see in elasticsearch)
• lots of messages in the aws s3 bucket for it to index but it hasn't continued

And I have no idea what to do / how to debug this

[Mahesh_Guntumadugu]

There should be Error log , otherwise try restarting your logstash and see if you get any reason for failure in log file. Still if you don't find any ERROR try running logstash with debug option and see the logs(bin\logstash --debug -f "path to your conf file")