Log visible in Argo but missing from Elasticsearch – only received last month briefly

Elastic Search
1

my app is deployed on K8s and managed via ArgoCD. Logs are collected using Fluentd (as a DaemonSet) and sent to Elasticsearch. Fluentd reads logs directly from container stdout/stderr (/var/log/containers/*.log) and the log format is JSON.

Both Fluentd and Elasticsearch configurations have not been modified recently.

Experiencing a situation where:

  • Logs are clearly visible in Argo, for example:
Properties.RequestPath: "/apple/api/v1/charge"
AND MessageTemplate: "HTTP {RequestMethod} {RequestPath} responded {StatusCode} in {Elapsed:0.0000} ms"
AND NOT Properties.ResponseBody.responseCode: (APPROVED, DECLINED_ACCOUNT_CLOSED, DECLINED_ACCOUNT_ON_HOLD, DECLINED_AMOUNT_EXCEEDED_UPPER_LIMIT, DECLINED_INSUFFICIENT_FUNDS)

Already received a few of these logs in Elasticsearch last month, but not before or after that period.
Fluentd is not using a parser plugin (e.g., @type json), so raw JSON is being sent.
There are no clear errors in Fluentd or Elasticsearch logs .
Due to permission restrictions, I cannot use Kibana Dev Tools to query index mappings or check recent ingested documents

I was wondering :sad_but_relieved_face:

  1. Some fields contain JSON strings (e.g., Properties.ResponseBodyStr). Could this impact proper ingestion into Elasticsearch?
  2. Is it possible for Elasticsearch to silently reject documents without Fluentd logging an error? How can we make error detection more visible?
  3. Given that some logs were successfully ingested last month, but not anymore despite the same format, what could be the possible causes? (e.g., mapping changes, document structure too deep, ES silently dropping data?)

Thanks for any help or suggestions :smiling_face_with_tear: