On our openshift clusters we have a wide range of applications and workloads running, which write logs in different ways.
If some developers write logs in json we want to enable them to annotate their workloads and to indicate that they want to have their json logs parsed into fields. At the same time, we don't want to know and to be involved in this configuration. Therefore we want to use hint based autodiscover. As far as I understand this is exactly what it is made for.
The reality is just: It does happen that some pods are still annotated with json annotations, but do not write json logs anymore for some reason. This results in spamming errors:
ERROR [reader_json] readjson/json.go:74 Error decoding JSON
You can suppress this error by giving:
json.ignore_decoding_error: true but it seems that once you set it, Filebeat starts to parse ALL json logs that are coming, even from pods that are not annotated. This leads to mapping explosions, mapping conflicts and crazy management overhead.
- Question: Do I understand correct that json.ignore_decoding_error: true, enabled json processor for all logs?
- Question: Is it possible to suppress
ERROR [reader_json] readjson/json.go:74 Error decoding JSONwithout enabling json processor for everything and just let hint based autodiscover work.