Fixing late logs into elastic

Hello there,

i would like to ask about my problem here. so here is the situation. my elastic, receive log from OCP. the problem is when the log is viewed from elastic, it doesn't appear to have updated. as in the image below, you can see the pattern of the log is interrupted at several points.

when I turn on auto refresh, the logs at the time I marked with square, it continues to grow and i can guarantee that the pod that logged that log continues to log every milliseconds. do you know why this is happen? does it have relation with resource usage on logstash?

fyi, i use ELK 7.17, Thanks

I'd suggest you to use filebeat instead of logstash to read the logs file since your's is TSD. Logstash is more useful when you want to read bulk file or rather static file OR when you want to perform some event manipulation using parsers (which again can be done using ES ingest pipelines).

Coming to your problem, do you have file rotation enabled either at application or host level? I've seen OCP clusters with CRIO nodes which perform log rotation by default at every 50M log file size. On other POV, is your application terminating or killed or restarting frequently ?

i already used filebeat by the way, it is in the form of a pod.

yes, file rotation is on at application level

No, from the start of this case I have never restarted the log source pod

OK, how frequent is log file rotation, what are the specs and how the file is being rotated? See: Log rotation results in lost or duplicate events | Filebeat Reference [master] | Elastic

file rotation will run once every 15 minutes. max file size is 50MB and keep files is 5 files

OK, there are some considerations for filebeat when it comes to log rotation, have you read the documentation that I shared earlier ? Also, when files are rotated, are they being archived as well into .gz or some other format, in which case beats wont be able to read the files. In addition, do the rotated files follow same naming as current log file ?
Can you please share your filebeat config to better understand your setup ?

yes, they will be archived with different name. for filebeat config, will follow soon

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.