Hi All
I am currently using 9 nodes Elasticsearch cluster in that while indexing the data.their is 5 to 10 minutes of time delay is happening. Input data EPS is about 2000 and there is no data is within the queue. How can I eliminate this delay?
Where is the data coming from? How are you ingesting it? What does the flow from raw data to Elasticsearch look like?
Hi Christian
i am sending the data from an automated shipper.the logs are in json format.I am ingesting the data through logstash.the raw data send to the logstash then to Elastic cluster from there to UI
Have you identified where the delay comes from? Logstash by default sets the @timestamp field to the time the event is received, so you could compare this to the event timestamp to see if the delay occurs there or whether it is later. If the delay is later, have you perhaps got a custom refresh interval set for your indices?
no i haven't checked on that.but there is no queue is happening with in threat pool.i can see the delay through the UI. while close checking the time stamp with the current machine time i am seeing the delay.
Try to identify where the delay occurs. Without that it is hard to speculate. How are you seeing the delay in the UI?
The attached @timestamp is showing that 5 minutes delay from the actual time.
sure i am trying to identify the issue
Can you show a sample document? What does your Logstash config look like? What type of shipper is sending data to Logstash? Are your hosts using NTP to synchronise time so there is no time drift that could contribute?
If the time difference between the event and the receive time stamp in Logstash is substantial, it could indicate an issue with the shipper, time drift between hosts or Logstash resource constraints.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.