Missing certain log entries (json from k8s using filebeat)

[geoaxis (Shahzada Hatim)]

I am a consumer of an elastic stack (filebeat, es, kibana) that gets structured json logs from a kubernetes cluster (writing to stdout). The stack is slef hosted by a group at my workplace and unfortunately it does not get the required attention (i.e disks fill up, rouge services spam the logs)

Some thing has happened in the last few days after an incident (where spamming indexes had to be deleted) and after that a set of log entries from my app have stopped appearing in Kibana. Upon inspection i can see that i can get those logs when i do kubectl logs. Other logs that come before in a transaction, from same pod/app/ingest method are working fine.

I have observed that one of the keys of the structured log has conflict in indexes (it appearss as text, keyword and object). Can it be that there is a conflict and a whole entry gets ignored.

I checked the type of the field and it is text, ignore_above: 1024 and the text i get in raw logs is never longer than 500. So that may not be an issue.

What could have changed?

I have been scratching my head over this for 10 days now. Any pointers would be greatly appreciated

[weltenwort (Felix Stürmer)]

Hi @geoaxis,

the type conflict you describe could indeed cause the ingestion to fail if Elasticsearch can't coerce the value into the corresponding field type determined by the mapping. That field type could either be determined explicitly and statically in an index template or dynamically when the first document with that field is indexed by Elasticsearch. In the latter case ES might incorrectly guess at the type. If a conflict happens ES should log it.

Could you check for such a ingestion failure? It might also be telling to see the index templates and mappings of the relevant indices and some example documents.

[geoaxis (Shahzada Hatim)]

Thank you, it was indeed the problem. The type of the structured argument changed the index pattern. I took away the error argument and rest of log entry started showing up. I will now eliminate the offending sources (from my code) and wait for offending indexes to be deleted.