I have a server with logstash, elasticsearch and kibana, and my index rating is about 4,500/s (107.4m documents in about 14 hours)
The problem is I'm having a lot of delay between the syslogs events and the elasticsearch indexed event. The delay is about two hours.
I'm indexing bluecoat logs, and I'm parsing the logs in logstash with grok and csv.
The server has 12 CPU with 32 GB of RAM. The storage is NFS, and I think that could be the problem, but I don't know how to see, if thats the problem certainly.
Another important detail, is in XPack monitoring I can see the traffic graphs with blank spaces, I mean, it seems elasticsearch stops indexing for one second.
How could I troubleshoot this problem? Is there any tunning option to configure in elasticsearch? (I disabled replicas and I change the refresh_interval to 10s)
Thanks in advance!