# Data delay in ELK **URL:** \ **Category:** Logstash\ **Created:** [January 23, 2017, 11:24am UTC](https://discuss.elastic.co/t/data-delay-in-elk/72472 "2017-01-23T11:24:22Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![sundar1](https://avatars.discourse-cdn.com/v4/letter/s/8dc957/32.png) [@sundar1](https://discuss.elastic.co/u/sundar1)\ **Post date:** [January 23, 2017, 11:24am UTC](https://discuss.elastic.co/t/data-delay-in-elk/72472/1 "2017-01-23T11:24:22Z")
Hi, Everyday we are getting close to 2 TB data and per sec 70K events from Kafka to ELK. The below are my ELK and hardware setup. 15 logstash servers(36 GM RAM, 6 CPU) 25 data nodes(36 GM RAM, 6 CPU) and each data node has 2 TB storage. We are getting data delay almost 2 days. Could you please suggest to me is any issues with my configurations and hardware. Regards Sundar ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [January 23, 2017, 11:36am UTC](https://discuss.elastic.co/t/data-delay-in-elk/72472/2 "2017-01-23T11:36:33Z")
Have you verified whether Elasticsearch or Logstash is limiting throughput? If so, how did you go about doing so? ---
**Author:** ![sundar1](https://avatars.discourse-cdn.com/v4/letter/s/8dc957/32.png) [@sundar1](https://discuss.elastic.co/u/sundar1)\ **Post date:** [January 23, 2017, 1:19pm UTC](https://discuss.elastic.co/t/data-delay-in-elk/72472/3 "2017-01-23T13:19:22Z")
Thanks Christian! I have tested 6k throughput from Kafka to logstash server but not tested throughput from logstash to elasticsearch .Is any way to test? ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [January 23, 2017, 1:38pm UTC](https://discuss.elastic.co/t/data-delay-in-elk/72472/4 "2017-01-23T13:38:41Z")
If each Logstash instance is able to process in excess of 6k events per second (90k per second in total) with all the filters present but without outputting to Elasticsearch, it sounds like either Elasticsearch or the Elasticsearch output plugin could be the bottleneck. Which version of Logstash are you using? How have you configured your Elasticsearch output plugin(s)? ---
**Author:** ![sundar1](https://avatars.discourse-cdn.com/v4/letter/s/8dc957/32.png) [@sundar1](https://discuss.elastic.co/u/sundar1)\ **Post date:** [January 23, 2017, 2:31pm UTC](https://discuss.elastic.co/t/data-delay-in-elk/72472/5 "2017-01-23T14:31:28Z")
I'm using Logstash 2.4 and elasticsearch is 5.0.2 version. output { elasticsearch { codec =\> avro { schema\_uri =\> "/apps/schema/rocana3.schema" } hosts =\> "[http://es-uat-rtp-master-ltm.xxx.com:9200/](http://es-uat-rtp-master-ltm.xxx.com:9200/)" index =\> "logstash-applogs-%{+YYYY.MM.dd}-1" workers =\> 6 } } ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [January 23, 2017, 3:20pm UTC](https://discuss.elastic.co/t/data-delay-in-elk/72472/6 "2017-01-23T15:20:00Z")
I would expect each Logstash node to be able to connect to all data nodes in order to spread the load, but it looks like each Logstash node is sending all traffic to a single node (which also based on the name seems to be a master node). Is traffic evenly spread across the cluster? ---
**Author:** ![sundar1](https://avatars.discourse-cdn.com/v4/letter/s/8dc957/32.png) [@sundar1](https://discuss.elastic.co/u/sundar1)\ **Post date:** [January 24, 2017, 10:58am UTC](https://discuss.elastic.co/t/data-delay-in-elk/72472/7 "2017-01-24T10:58:04Z")
Thanks for the reply and removed ltm url from host and given all data nodes in the host. host =\> ["host1","host2",............"host25"] even not getting more data to the elasticsearch data node. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [January 24, 2017, 11:08am UTC](https://discuss.elastic.co/t/data-delay-in-elk/72472/8 "2017-01-24T11:08:24Z")
What does the resource utilisation look like on the Elasticsearch nodes? Do you see high CPU usage and/or high iowait? Is there anything in the Elasticsearch logs indicating e.g. long GC or merge throttling? ---
**Author:** ![sundar1](https://avatars.discourse-cdn.com/v4/letter/s/8dc957/32.png) [@sundar1](https://discuss.elastic.co/u/sundar1)\ **Post date:** [January 24, 2017, 11:48am UTC](https://discuss.elastic.co/t/data-delay-in-elk/72472/9 "2017-01-24T11:48:33Z")
Seems CPU and IO is fine. No issues for GC. Please find below are my ELK config hope it will help you to understand my configuration and suggest if anything is wrong **Linux Infrastructure for logstash, ES and Kibana** Hardware 6 CPU / 32 GB RAM Operating System Oracle Enterprise Linux 6 FID16a 2X-Large **Logstash :** input { kafka { zk\_connect=\>"kafka1:2181,kafka2:2181,kafka3:2181,kafka4:2181,kafka5:2181" white\_list =\> "applogs" group\_id =\> "logstash-mng-applogs-uat-rtp" codec =\> avro { schema\_uri =\> "/apps/schema/rocana3.schema" } } } filter { de\_dot { nested =\> true } date { match =\> ["ts","UNIX\_MS"] target =\> "@timestamp" timezone =\> "America/New\_York" } ruby { code =\> " event['ingest\_time'] = DateTime.now.strftime('%Q'); event['ingest\_delay'] = (1000 \* (Time.now.to\_f - event['@timestamp'].to\_f)).round(0); " } } output { elasticsearch { hosts =\> ["[es-uat-rtp-data-1.xxx.com:9200](http://es-uat-rtp-data-1.xxx.com:9200)","[es-uat-rtp-data-2.xxx.com:9200](http://es-uat-rtp-data-2.xxx.com:9200)","[es-uat-rtp-data-3.xxx.com:9200](http://es-uat-rtp-data-3.xxx.com:9200)","[es-uat-rtp-data-4.xxx.com:9200](http://es-uat-rtp-data-4.xxx.com:9200)","[es-uat-rtp-data-5.xxx.com:9200](http://es-uat-rtp-data-5.xxx.com:9200)","[es-uat-rtp-data-6.xxx.com:9200](http://es-uat-rtp-data-6.xxx.com:9200)","[es-uat-rtp-data-7.xxx.com:9200](http://es-uat-rtp-data-7.xxx.com:9200)","[es-uat-rtp-data-8.xxx.com:9200](http://es-uat-rtp-data-8.xxx.com:9200)","[es-uat-rtp-data-9.xxx.com:9200](http://es-uat-rtp-data-9.xxx.com:9200)","[es-uat-rtp-data-10.xxx.com:9200](http://es-uat-rtp-data-10.xxx.com:9200)","[es-uat-rtp-data-11.xxx.com:9200](http://es-uat-rtp-data-11.xxx.com:9200)","[es-uat-rtp-data-12.xxx.com:9200](http://es-uat-rtp-data-12.xxx.com:9200)","[es-uat-rtp-data-13.xxx.com:9200](http://es-uat-rtp-data-13.xxx.com:9200)","[es-uat-rtp-data-14.xxx.com:9200](http://es-uat-rtp-data-14.xxx.com:9200)","[es-uat-rtp-data-15.xxx.com:9200](http://es-uat-rtp-data-15.xxx.com:9200)","[es-uat-rtp-data-16.xxx.com:9200](http://es-uat-rtp-data-16.xxx.com:9200)","[es-uat-rtp-data-17.xxx.com:9200](http://es-uat-rtp-data-17.xxx.com:9200)","[es-uat-rtp-data-18.xxx.com:9200](http://es-uat-rtp-data-18.xxx.com:9200)","[es-uat-rtp-data-19.xxx.com:9200](http://es-uat-rtp-data-19.xxx.com:9200)","[es-uat-rtp-data-20.xxx.com:9200](http://es-uat-rtp-data-20.xxx.com:9200)","[es-uat-rtp-data-21.xxx.com:9200](http://es-uat-rtp-data-21.xxx.com:9200)","[es-uat-rtp-data-22.xxx.com:9200](http://es-uat-rtp-data-22.xxx.com:9200)","[es-uat-rtp-data-23.xxx.com:9200](http://es-uat-rtp-data-23.xxx.com:9200)","[es-uat-rtp-data-24.xxx.com:9200](http://es-uat-rtp-data-24.xxx.com:9200)","[es-uat-rtp-data-25.xxx.com:9200](http://es-uat-rtp-data-25.xxx.com:9200)"] index =\> "logstash-applogs-%{+YYYY.MM.dd}-1" workers =\> 6 } } **Master Node ES :-** [cluster.name](http://cluster.name): sei-elk-uat-rtp [node.name](http://node.name): sundar-master-01 node.master: true node.data: false path.data: /apps/masterES/data path.logs: /apps/masterES/logs bootstrap.memory\_lock: true network.host: 01.02.03.04 http.port: 9200 discovery.zen.ping.unicast.hosts: ["master1 ip","master2 ip","master3 ip"] discovery.zen.minimum\_master\_nodes: 2 http.cors.enabled: true http.cors.allow-origin: "\*" **Data Node ES :-** [cluster.name](http://cluster.name): sei-elk-uat-rtp [node.name](http://node.name): sundar-data-01 node.master: false node.data: true path.data: /apps/dataES1/data path.logs: /apps/dataES1/logs discovery.zen.ping.unicast.hosts: ["master1 ip","master2 ip","master3 ip"] network.host: 05.06.07.08 http.port: 9200 bootstrap.memory\_lock: true **Client ES:-** [cluster.name](http://cluster.name): sei-elk-uat-rtp [node.name](http://node.name): sundar-client-01 node.master: false node.data: false path.data: /apps/clientES/data path.logs: /apps/clientES/logs network.host: 10.138.000.00 http.port: 9200 discovery.zen.ping.unicast.hosts: ["master1 ip","master2 ip","master3 ip"] ---
**Author:** ![sundar1](https://avatars.discourse-cdn.com/v4/letter/s/8dc957/32.png) [@sundar1](https://discuss.elastic.co/u/sundar1)\ **Post date:** [January 25, 2017, 3:02pm UTC](https://discuss.elastic.co/t/data-delay-in-elk/72472/10 "2017-01-25T15:02:58Z")
Today we have created close to 50 data nodes(2TB storage),3 masters and 3 clients. still we are not getting more index rate. the screen has monitoring details. Could you please me if any wrong settings. ![](https://us1.discourse-cdn.com/elastic/original/2X/1/1e4a6f061dc1995a702c5f01231c7d1d745f7071.png) ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [January 25, 2017, 3:23pm UTC](https://discuss.elastic.co/t/data-delay-in-elk/72472/11 "2017-01-25T15:23:25Z")
If you have doubled the size of the cluster, traffic is distributed across all nodes and the shards being indexed into are spread across node and you are still not seeing any performance improvement, it is quite possible that Logstash after all is limiting throughput. The Kafka input has a range of configuration parameters that you can tune for performance, e.g. consumer\_threads. It may be worthwhile tuning this, but I have personally not done it so can not really give any advice on this. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 22, 2017, 3:23pm UTC](https://discuss.elastic.co/t/data-delay-in-elk/72472/12 "2017-02-22T15:23:44Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.