We are having the following error on our ELK5 Stack:
[2018-07-04T09:25:24,747][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 500 ({"type"=>"class_cast_exception", "reason"=>"class_cast_exception: null"})
[2018-07-04T09:25:24,748][INFO ][logstash.outputs.elasticsearch] Retrying individual bulk actions that failed or were rejected by the previous bulk request. {:count=>1}
[2018-07-04T09:25:26,763][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 500 ({"type"=>"class_cast_exception", "reason"=>"class_cast_exception: null"})
[2018-07-04T09:25:26,763][INFO ][logstash.outputs.elasticsearch] Retrying individual bulk actions that failed or were rejected by the previous bulk request. {:count=>1}
[2018-07-04T09:25:30,772][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 500 ({"type"=>"class_cast_exception", "reason"=>nil})
[2018-07-04T09:25:30,773][INFO ][logstash.outputs.elasticsearch] Retrying individual bulk actions that failed or were rejected by the previous bulk request. {:count=>1}
[2018-07-04T09:25:38,789][INFO ][logstash.outputs.elasticsearch] retrying failed action with response code: 500 ({"type"=>"class_cast_exception", "reason"=>nil})
[2018-07-04T09:25:38,790][INFO ][logstash.outputs.elasticsearch] Retrying individual bulk actions that failed or were rejected by the previous bulk request. {:count=>1}
We are using the default logstash template and the messages we are processing are the following:
-
Message1: "{"EventSeverity":"Error","Timestamp":"2018-07-03T15:29:51.3483049Z","Msg":{"Message":"Error","Data":{"Parameters":[{"Key":"key1","Value":"value"},{"Key":"key2","Value":"2018-07-03T15:29:19.141Z"}],"HostName":"vm1"}}}
{"EventSeverity":"Error","Timestamp":"2018-07-03T15:29:51.3483049Z","Msg":{"Message":"Error","Data":{"Parameters":[{"Key":"key1","Value":"value"}],"HostName":"vm1"}}}" -
Message2: "{"EventSeverity":"Error","Timestamp":"2018-07-03T15:29:51.3483049Z","Msg":{"Message":"Error","Data":{"Parameters":[{"Key":"key1","Value":"value"}],"HostName":"vm1"}}}"
Both messages are json, so we are using logstash json parser. If we send the Message2 first, the index is created and the message is inserted on elasticsearch. If we then process Message1, it's also inserted on elasticsearch. However, if we send Message1 first, we get the 500 response code and class_cast_exception error. We keep seeing the log appearing for a lot of time which seems to indicate that logstash keeps sending the log to elasticsearch and stop processing/sending news logs until we restart it. Can you tell us why we are getting the class_cast_exception log and how can you "tell" logstash to stop send the message to elasticsearch and discard it after X tries?