# Slow ingestion using Logstash Filebeat output with Kubernetes autodiscovery **URL:** \ **Category:** Logstash\ **Created:** [April 11, 2019, 5:28pm UTC](https://discuss.elastic.co/t/slow-ingestion-using-logstash-filebeat-output-with-kubernetes-autodiscovery/176466 "2019-04-11T17:28:18Z")\ **Posts on this page:** 3\ **Page:** 1
**Author:** ![bagratte](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bagratte/32/37155_2.png) [@bagratte](https://discuss.elastic.co/u/bagratte)\ **Post date:** [April 11, 2019, 5:28pm UTC](https://discuss.elastic.co/t/slow-ingestion-using-logstash-filebeat-output-with-kubernetes-autodiscovery/176466/1 "2019-04-11T17:28:19Z")
I am using Elastic Stack for harvesting logs of a Kubernetes cluster. Elastic Stack is installed in the same cluster and I'm using Filebeat's Kubernetes autodiscover. Elastic Stack consists of an Elasticsearch cluster with 5 data only nodes, 3 dedicated master nodes and 2 client (coordinating) nodes; 2 Logstash instances using multiple pipelines; and 16 Filebeat instances, one on each Kubernetes node (daemonset). The Kubernetes cluster generates some fair amount of logs, peaking at a rate of around 200.000 log entries per 15 minutes (which makes it, what, 222 events/sec, to use more proper rate units?). When I start Elastic Stack for the first time, logs get ingested fine and quickly. Then, after an hour or so, rates in all of the components drop. That is, there is a drop in Filebeat events rate, in Logstash events received and emitted rates and in Elasticsearch indexing rate. There are errors in Filebeat logs too: ```auto 2019-04-11T16:28:34.768Z ERROR [autodiscover] cfgfile/list.go:96 Error creating runner from config: Can only start an input when all related states are finished: {Id:4200643-66306 Finished:false Fileinfo:0xc000d1bba0 Source:/var/lib/docker/containers/7858e93927986b4630903a04614e646889b3f32ebe38b84550c73247a0da0c38/7858e93927986b4630903a04614e646889b3f32ebe38b84550c73247a0da0c38-json.log Offset:0 Timestamp:2019-04-11 16:26:05.646027461 +0000 UTC m=+1337.750836655 TTL:-1ns Type:docker Meta:map[] FileStateOS:4200643-66306} ``` and sometimes ```auto 2019-04-11T15:32:52.622Z ERROR logstash/async.go:256 Failed to publish events caused by: read tcp 100.102.132.75:57572->100.110.196.158:5044: i/o timeout 2019-04-11T15:32:52.622Z ERROR logstash/async.go:256 Failed to publish events caused by: read tcp 100.102.132.75:57572->100.110.196.158:5044: i/o timeout 2019-04-11T15:32:52.636Z ERROR logstash/async.go:256 Failed to publish events caused by: client is not connected 2019-04-11T15:32:53.966Z ERROR [autodiscover] cfgfile/list.go:96 Error creating runner from config: Can only start an input when all related states are finished: {Id:3156173-66306 Finished:false Fileinfo:0xc000c74ea0 Source:/var/lib/docker/containers/2780263029cdda7e1fe19d34c77501207d70559d45501b0c57ed14930ea6c212/2780263029cdda7e1fe19d34c77501207d70559d45501b0c57ed14930ea6c212-json.log Offset:0 Timestamp:2019-04-11 15:32:02.946308974 +0000 UTC m=+544.111177032 TTL:-1ns Type:docker Meta:map[] FileStateOS:3156173-66306} 2019-04-11T15:32:54.083Z ERROR pipeline/output.go:121 Failed to publish events: client is not connected 2019-04-11T15:32:54.083Z INFO pipeline/output.go:95 Connecting to backoff(async(tcp://logstash-1.logstash:5044)) 2019-04-11T15:32:54.092Z INFO pipeline/output.go:105 Connection to backoff(async(tcp://logstash-1.logstash:5044)) established ``` but fiddling with `output.logstash.bulk_max_size` in Filebeat config **appears** to be solving these network related errors. Autodiscover errors continue. There is also a few of ```auto 2019-04-11T16:26:25.595Z ERROR kubernetes/watcher.go:266 kubernetes: Watching API error context canceled, ignoring event and moving to most recent resource version ``` but I've seen these in perfectly functioning Elastic Stacks in other Kubernetes clusters too, so these are probably irrelevant. No errors in Logstash or Elasticsearch logs. Logstashes report around 10% CPU usage in the beginning, when ingestion is fine, then it drops too to around 1-2%. **If I use `output.elasticsearch` in Filebeat, pointing to a loadbalancer (Kubernetes service) targeting the Elasticsearch client nodes, none of these errors occur. Nore the drop in ingestion. Logs keep ending up in Elasticsearch fine for days and days.** All of the components of Elastic Stack are currently version 7.0.0 but the problem occurs also with version 6. Any ideas of where shall I look for tuning Elastic Stack to keep up with my logs? I've tried lots of suggestions brought up in other threads on Internet (and this forum), like tweaking `pipeline.workers` in Logstash or `output.logstash.worker` in Filebeat or `pipeline.batch.size` in Logstash with no apparent success. ---
**Author:** ![bagratte](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/bagratte/32/37155_2.png) [@bagratte](https://discuss.elastic.co/u/bagratte)\ **Post date:** [May 6, 2019, 11:04am UTC](https://discuss.elastic.co/t/slow-ingestion-using-logstash-filebeat-output-with-kubernetes-autodiscovery/176466/2 "2019-05-06T11:04:40Z")
The bottleneck was the disk IOPS of Logstash persisted queue. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 3, 2019, 11:13am UTC](https://discuss.elastic.co/t/slow-ingestion-using-logstash-filebeat-output-with-kubernetes-autodiscovery/176466/3 "2019-06-03T11:13:15Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.