# Data delay writing to ES

**URL:** <https://discuss.elastic.co/t/data-delay-writing-to-es/118139>\
**Category:** Logstash\
**Created:** [February 2, 2018, 1:36am UTC](https://discuss.elastic.co/t/data-delay-writing-to-es/118139 "2018-02-02T01:36:29Z")\
**Posts on this page:** 7\
**Page:** 1

<div class="post-metadata">

**Author:** ![antony.y](https://avatars.discourse-cdn.com/v4/letter/a/91b2a8/32.png) [@antony.y](https://discuss.elastic.co/u/antony.y)\
**Post date:** [February 2, 2018, 1:36am UTC](https://discuss.elastic.co/t/data-delay-writing-to-es/118139/1 "2018-02-02T01:36:29Z")

</div>

We are currently using logstash collect the netflow from our routers, we expected a real-time data,but now we found that we can only see the data of yesterday, that means the data delayed for one day, and we check our persiste queue, the data in the queue is only 4G , is there anyway to solve the data delay ?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [February 2, 2018, 10:21am UTC](https://discuss.elastic.co/t/data-delay-writing-to-es/118139/2 "2018-02-02T10:21:58Z")

</div>

If the PQ is growing, it would seem like either Logstash or your downstream systems are not able to keep up with the flow. What does your configuration look like? Where are you sending data? What throughput are you seeing?

---

<div class="post-metadata">

**Author:** ![antony.y](https://avatars.discourse-cdn.com/v4/letter/a/91b2a8/32.png) [@antony.y](https://discuss.elastic.co/u/antony.y)\
**Post date:** [February 3, 2018, 8:18am UTC](https://discuss.elastic.co/t/data-delay-writing-to-es/118139/3 "2018-02-03T08:18:25Z")

</div>

The configuration of my logstash is like:

input {  
udp {  
type =\> "netflow"  
port =\> \<%= setting("var.input.udp.port", 2055) %\>  
codec =\> netflow {  
versions =\> [5,9]  
}  
add\_field =\> {"datacenter" =\> "Eastern"}  
}  
}

output {  
\<%= elasticsearch\_output\_config() %\>  
}

I am now using the netflow module in logstash immediately,  
I sent my netflow from our network devices  
I am now seeing the realtime traffic,but now (16:17) i can only see the data before 15:00

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/9/2/92b4aed06a8f8be22226fad608de6fccd893258b.png)

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [February 3, 2018, 8:52am UTC](https://discuss.elastic.co/t/data-delay-writing-to-es/118139/4 "2018-02-03T08:52:52Z")

</div>

What indexing throughput are you seeing in Elasticsearch? What is the specification of your Elasticsearch cluster? Do you have monitoring installed?

---

<div class="post-metadata">

**Author:** ![antony.y](https://avatars.discourse-cdn.com/v4/letter/a/91b2a8/32.png) [@antony.y](https://discuss.elastic.co/u/antony.y)\
**Post date:** [February 3, 2018, 9:55am UTC](https://discuss.elastic.co/t/data-delay-writing-to-es/118139/5 "2018-02-03T09:55:15Z")

</div>

The index in my es is like "netflow-idc-2018.02.03",in my ealsticsearch we have 9 nodes(including 4 data nodes, 2 master nodes,2 coordinating nodes),And now we are using kibana monitor the health of my es cluster,but it seems that all the things are normal through the monitoring

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [February 3, 2018, 9:59am UTC](https://discuss.elastic.co/t/data-delay-writing-to-es/118139/6 "2018-02-03T09:59:17Z")

</div>

What is the indexing throughput? What is the specification of the hosts the data nodes are running on?

Also, having 2 master nodes is bad - you should always look to have 3.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 3, 2018, 9:59am UTC](https://discuss.elastic.co/t/data-delay-writing-to-es/118139/7 "2018-03-03T09:59:53Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.