# Huge delay in logs to ES from Kafka/Logstash **URL:** \ **Category:** Elasticsearch\ **Created:** [July 6, 2020, 3:16am UTC](https://discuss.elastic.co/t/huge-delay-in-logs-to-es-from-kafka-logstash/239948 "2020-07-06T03:16:38Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![biswajit\_list](https://avatars.discourse-cdn.com/v4/letter/b/c77e96/32.png) [@biswajit\_list](https://discuss.elastic.co/u/biswajit_list)\ **Post date:** [July 6, 2020, 3:16am UTC](https://discuss.elastic.co/t/huge-delay-in-logs-to-es-from-kafka-logstash/239948/1 "2020-07-06T03:16:38Z")
New for ELK Have 2 Kafka , 2 Logstash , 3 master ES and 2 Data nodes my logstash config is input { kafka { bootstrap\_servers =\> "kafka01:9092, kafka02:9092" topics =\> ["filebeat"] codec =\> json heartbeat\_interval\_ms =\> "1000" poll\_timeout\_ms =\> "10000" session\_timeout\_ms =\> "120000" request\_timeout\_ms =\> "130000" consumer\_threads =\> 40 } } filter { mutate { add\_field =\> {"[@metadata][index]" =\> "%{[kafka][topic]}"} } } output { elasticsearch { hosts =\> ["esmas01:9200", "esmas02:9200", "esmas03:9200"] index =\> "logstash-%{[@metadata][index]}-%{+YYYY.MM.dd}" } stdout { codec =\> rubydebug } } I could See logs are reaching Kafka , Looks like Logstash consumption delay or Outpul delay to ES Can some one help . TIA ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 6, 2020, 5:12am UTC](https://discuss.elastic.co/t/huge-delay-in-logs-to-es-from-kafka-logstash/239948/2 "2020-07-06T05:12:17Z")
The first thing to do is remove the stdout output. Elasticsearch can often be the bottleneck so it makes sense to check it first. It would help if you could tell us the specification of the Elasticsearch nodes ( number of CPU cotes, RAM and size and type of storage). How much data do you have in the cluster? How much are you writing per day? How many indices and shards are you indexing into? ---
**Author:** ![biswajit\_list](https://avatars.discourse-cdn.com/v4/letter/b/c77e96/32.png) [@biswajit\_list](https://discuss.elastic.co/u/biswajit_list)\ **Post date:** [July 6, 2020, 2:20pm UTC](https://discuss.elastic.co/t/huge-delay-in-logs-to-es-from-kafka-logstash/239948/3 "2020-07-06T14:20:59Z")
Thanks Christian for helping Removed stdout , found some improvement ES master nodes are 3 core / 16 GB , ES Data node 3 core / 12 GB . storage is attached to Data node is San based Data is around 1.1 TB per Data nodes `curl -X GET "localhost:9200/_cat/shards?h=index,shard,prirep,state,unassigned.reason&pretty" .monitoring-kibana-7-2020.07.06 0 p STARTED .monitoring-kibana-7-2020.07.06 0 r STARTED .kibana_1 0 p STARTED .kibana_1 0 r STARTED logstash-%{[kafka][topic]}-2020.06.26 0 p STARTED logstash-%{[kafka][topic]}-2020.06.26 0 r STARTED logstash-%{[kafka][topic]}-2020.06.28 0 p STARTED logstash-%{[kafka][topic]}-2020.06.28 0 r STARTED .apm-agent-configuration 0 p STARTED .apm-agent-configuration 0 r STARTED logstash-%{[kafka][topic]}-2020.06.15 0 p STARTED logstash-%{[kafka][topic]}-2020.06.15 0 r STARTED .monitoring-es-7-2020.07.05 0 p STARTED .monitoring-es-7-2020.07.05 0 r STARTED logstash-%{[kafka][topic]}-2020.06.24 0 p STARTED logstash-%{[kafka][topic]}-2020.06.24 0 r STARTED logstash-%{[kafka][topic]}-2020.07.01 0 p STARTED logstash-%{[kafka][topic]}-2020.07.01 0 r STARTED logstash-%{[kafka][topic]}-2020.06.19 0 p STARTED logstash-%{[kafka][topic]}-2020.06.19 0 r STARTED logstash-%{[kafka][topic]}-2020.06.29 0 p STARTED logstash-%{[kafka][topic]}-2020.06.29 0 r STARTED .async-search 0 p STARTED .async-search 0 r STARTED logstash-%{[kafka][topic]}-2020.06.21 0 p STARTED logstash-%{[kafka][topic]}-2020.06.21 0 r STARTED logstash-%{[kafka][topic]}-2020.06.18 0 p STARTED logstash-%{[kafka][topic]}-2020.06.18 0 r STARTED logstash-%{[kafka][topic]}-2020.06.30 0 p STARTED logstash-%{[kafka][topic]}-2020.06.30 0 r STARTED logstash-%{[kafka][topic]}-2020.06.25 0 p STARTED logstash-%{[kafka][topic]}-2020.06.25 0 r STARTED logstash-%{[kafka][topic]}-2020.06.22 0 p STARTED logstash-%{[kafka][topic]}-2020.06.22 0 r STARTED logstash-%{[kafka][topic]}-2020.06.23 0 p STARTED logstash-%{[kafka][topic]}-2020.06.23 0 r STARTED logstash-%{[kafka][topic]}-2020.06.17 0 p STARTED logstash-%{[kafka][topic]}-2020.06.17 0 r STARTED logstash-%{[kafka][topic]}-2020.06.20 0 p STARTED logstash-%{[kafka][topic]}-2020.06.20 0 r STARTED logstash-%{[kafka][topic]}-2020.07.04 0 p STARTED logstash-%{[kafka][topic]}-2020.07.04 0 r STARTED .kibana_task_manager_1 0 p STARTED .kibana_task_manager_1 0 r STARTED logstash-%{[kafka][topic]}-2020.07.03 0 p STARTED logstash-%{[kafka][topic]}-2020.07.03 0 r STARTED logstash-%{[kafka][topic]}-2020.06.14 0 p STARTED logstash-%{[kafka][topic]}-2020.06.14 0 r STARTED logstash-%{[kafka][topic]}-2020.07.02 0 p STARTED logstash-%{[kafka][topic]}-2020.07.02 0 r STARTED .apm-custom-link 0 p STARTED .apm-custom-link 0 r STARTED logstash-%{[kafka][topic]}-2020.06.16 0 p STARTED logstash-%{[kafka][topic]}-2020.06.16 0 r STARTED kibana_sample_data_ecommerce 0 p STARTED kibana_sample_data_ecommerce 0 r STARTED .monitoring-es-7-2020.07.06 0 p STARTED .monitoring-es-7-2020.07.06 0 r STARTED logstash-%{[kafka][topic]}-2020.06.27 0 p STARTED logstash-%{[kafka][topic]}-2020.06.27 0 r STARTED .monitoring-kibana-7-2020.07.05 0 p STARTED .monitoring-kibana-7-2020.07.05 0 r STARTED ` I hope these information are sufficient to help me on this issue TIA ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [July 7, 2020, 5:22am UTC](https://discuss.elastic.co/t/huge-delay-in-logs-to-es-from-kafka-logstash/239948/4 "2020-07-07T05:22:37Z")
I have some additional questions: - What is the output of the [cluster stats API](https://www.elastic.co/guide/en/elasticsearch/reference/current/cluster-stats.html)? - What does CPU usage on the data nodes look like while you are indexing? - What does `iostat -x` give on the data nodes? - How did you arrive at the settings you are using for the Logstash kafka plugin? - Is there anything in the logs indicating slow or frequent GC? Are there any other warning or error messages, espacially on the data nodes? ---
**Author:** ![biswajit\_list](https://avatars.discourse-cdn.com/v4/letter/b/c77e96/32.png) [@biswajit\_list](https://discuss.elastic.co/u/biswajit_list)\ **Post date:** [July 7, 2020, 9:26am UTC](https://discuss.elastic.co/t/huge-delay-in-logs-to-es-from-kafka-logstash/239948/5 "2020-07-07T09:26:12Z")
Hi @Christian_Dahlqvist The issue seems to have got resolved by removing "stdout output" from logstash config . Thanks for the Support ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [August 4, 2020, 9:26am UTC](https://discuss.elastic.co/t/huge-delay-in-logs-to-es-from-kafka-logstash/239948/6 "2020-08-04T09:26:14Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.