How to detect and log 400 bad requests?

Elastic Stack Elasticsearch
1

Hello,

My apologies if asked before but I've been scouting the forums and the docs for hours trying to figure this out..

My setup:

Fluentd daemonset on Kubernetes collecting logs from containers and sending them to AWS Elasticsearch service by bulk API

Problem:

Sometimes apps write malformed logs, resulting in 400 bad requests (mapping conflicts). I have logs for fluentd but it only contains the response, so I can only see the error and the index name.

Is it possible to log request body+error on 400 errors in Elasticsearch? Payload will help me immensely figure out what apps are misbehaving

2

As far as I know it is not possible. To support this type of analysis, Logstash introduced a dead-letter queue to which incorrect entries can be written and later analysed. Here it is however the client that matches event with response and takes action.

1 Like
3

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.