I have trouble with ingesting from one of our logs. let me tell you the conditions first:
we have been ingesting our logs from OCP4 to Elastic through Logstash and generally, we have 2 sites of the Elastic cluster and 2 sites of the OCP4 cluster. so it's like OCP cluster A sent logs to elastic cluster A and so on
the logs that we ingest are from all existing namespace there
from an elastic point of view, there is one index that always looks delayed. let's say the name is Athena
if now is 17:15, and I pull out the data from the last 15 minutes. elastic will show data from 17:00 to 17:15. For normal data, the chart will filled by the log bar till the last seconds like this picture below
but for the Athena index, the log bar is still at 17:03. it still increasing, but it was far behind the times at that time. sometimes it's even worse. if I pull out data from the last 15 minutes, there's nothing. and this happens on both Elastic cluster
when I checked Athena's log directly from the pod, I found there was no delay at all. even the logs are printed every second.
These are index templates for the Athena index from both elastic clusters:
This strange behavior only happens at 1 index only on both clusters. idk why.
Can you help me with this?
if you have any questions to get a better understanding, please don't hesitate to ask me