Logging freezes every hour


(Kevin Ross) #1

Hi,

I have noticed in Kibana that every hour nearly on the hour logs stop going into elasticsearch for about 1 minute and then it seems to perhaps to a batch update as they all seem to find they way in as far as I can tell.

There is no jobs I run hourly which may explain it and both logstash and elasticsearch are running ok. I am running elasticsearch 2.1.1 and I don't recall seeing this before although it may be memory. I have a suspicion it is perhaps due to logstash memory as before there was an issue with it before it got set to 1GB where I would occasionally stop logging for hours - especially if using geoip. I have increased memory to 2GB and have 4 workers running.

I am wondering if there is any ideas of what it may be such as something logstash or elasticsearch does every hour which may explain it & also how to test this. Also how to I look at the logstash usage such as heap usage because now it does not connect as a node these stats seem to be invisble to me and I can't see this aspect of my logging?

Thank you very much


(Kevin Ross) #2

oh it may be useful to note the ELK stack is running on 1 box. 64GB RAM using 50% heap (for elasticsearch) which is set at 30GB and general RAM, CPU etc seems fine.


(Mark Walkom) #3

Try turning on GC logging for LS and see what's there?


(system) #4