Elasticsearch indexing slow

Elastic Stack Elasticsearch
1

Hi everyone!

I have a server with logstash, elasticsearch and kibana, and my index rating is about 4,500/s (107.4m documents in about 14 hours)

The problem is I'm having a lot of delay between the syslogs events and the elasticsearch indexed event. The delay is about two hours.

I'm indexing bluecoat logs, and I'm parsing the logs in logstash with grok and csv.

The server has 12 CPU with 32 GB of RAM. The storage is NFS, and I think that could be the problem, but I don't know how to see, if thats the problem certainly.

Another important detail, is in XPack monitoring I can see the traffic graphs with blank spaces, I mean, it seems elasticsearch stops indexing for one second.

How could I troubleshoot this problem? Is there any tunning option to configure in elasticsearch? (I disabled replicas and I change the refresh_interval to 10s)

Thanks in advance!
Regards

2

This is a single server? Not a clustered Elasticsearch? A single node can do a lot, but that might be part of the slowdown.

This is not recommended. In fact, do not ever do this. See our storage recommendations here.

This could be any of a few things.

  • Garbage collection in the JVM causes a pause in indexing
  • Segment merges cause a pause in indexing (because you only have one node)
  • Storage delays due to NFS (lock files and the like)
  • Yet other possibilities.

These are most likely, though. Your best bet to get to real-time indexing is to have more nodes in your Elasticsearch cluster with local SSD storage.

3

@theuntergeek Thanks so much for the info! I'll try to install another elasticsearch node (If someone gives me one server :stuck_out_tongue:). I'm afraid NFS is the only way I have to storage all data.

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.