# Kubernetes watch connection never close when inputreload create runner failed **URL:** \ **Category:** Beats\ **Tags:** filebeat\ **Created:** [May 8, 2019, 7:10am UTC](https://discuss.elastic.co/t/kubernetes-watch-connection-never-close-when-inputreload-create-runner-failed/180112 "2019-05-08T07:10:12Z")\ **Posts on this page:** 4\ **Page:** 1
**Author:** ![danielQ](https://avatars.discourse-cdn.com/v4/letter/d/85f322/32.png) [@danielQ](https://discuss.elastic.co/u/danielQ)\ **Post date:** [May 8, 2019, 7:10am UTC](https://discuss.elastic.co/t/kubernetes-watch-connection-never-close-when-inputreload-create-runner-failed/180112/1 "2019-05-08T07:10:12Z")
Hi, guys: filebeat run as daemonset in kubernetes cluster and set reload true, when config has changed and if reload create runner failed, it won't close kubernetes watch connection with kube-apiserver, and it will keep increasing the connections, at the end lead to kube-apiserver memory pressure . is it A already known problem or do i made some mistake? ```auto filebeat.config: inputs: # Mounted `filebeat-inputs` configmap: path: ${path.config}/inputs.d/*.json # Reload inputs configs as they change: reload.enabled: true modules: path: ${path.config}/modules.d/*.yml # Reload module configs as they change: reload.enabled: true processors: - add_cloud_metadata: cloud.id: ${ELASTIC_CLOUD_ID} cloud.auth: ${ELASTIC_CLOUD_AUTH} output.elasticsearch: hosts: ['${ELASTICSEARCH_HOST:elasticsearch}:${ELASTICSEARCH_PORT:9200}'] username: ${ELASTICSEARCH_USERNAME} password: ${ELASTICSEARCH_PASSWORD} ``` ```auto 2019-05-08T07:08:17.261Z ERROR [reload] cfgfile/list.go:104 Error creating runner from config: No paths were defined for input accessing config 2019-05-08T07:08:27.269Z ERROR [reload] cfgfile/list.go:104 Error creating runner from config: No paths were defined for input accessing config 2019-05-08T07:08:37.279Z ERROR [reload] cfgfile/list.go:104 Error creating runner from config: No paths were defined for input accessing config 2019-05-08T07:08:47.288Z ERROR [reload] cfgfile/list.go:104 Error creating runner from config: No paths were defined for input accessing config 2019-05-08T07:08:57.297Z ERROR [reload] cfgfile/list.go:104 Error creating runner from config: No paths were defined for input accessing config 2019-05-08T07:09:07.307Z ERROR [reload] cfgfile/list.go:104 Error creating runner from config: No paths were defined for input accessing config ``` ```auto tcp 0 0 192.168.1.86:47956 192.168.1.86:8080 ESTABLISHED 22052/filebeat tcp 0 0 192.168.1.86:54902 192.168.1.86:8080 ESTABLISHED 22052/filebeat tcp 0 0 192.168.1.86:47336 192.168.1.86:8080 ESTABLISHED 22052/filebeat tcp 0 0 192.168.1.86:43552 192.168.1.86:8080 ESTABLISHED 22052/filebeat tcp 0 0 192.168.1.86:37720 192.168.1.86:8080 ESTABLISHED 22052/filebeat tcp 0 0 192.168.1.86:49188 192.168.1.86:8080 ESTABLISHED 22052/filebeat tcp 0 0 192.168.1.86:53730 192.168.1.86:8080 ESTABLISHED 22052/filebeat tcp 0 0 192.168.1.86:60706 192.168.1.86:8080 ESTABLISHED 22052/filebeat tcp 0 0 192.168.1.86:56284 192.168.1.86:8080 ESTABLISHED 22052/filebeat tcp 0 0 192.168.1.86:49542 192.168.1.86:8080 ESTABLISHED 22052/filebeat tcp 0 0 192.168.1.86:56986 192.168.1.86:8080 ESTABLISHED 22052/filebeat tcp 0 0 192.168.1.86:55332 192.168.1.86:8080 ESTABLISHED 22052/filebeat tcp 0 0 192.168.1.86:55166 192.168.1.86:8080 ESTABLISHED 22052/filebeat tcp 0 0 192.168.1.86:49522 192.168.1.86:8080 ESTABLISHED 22052/filebeat tcp 0 0 192.168.1.86:49450 192.168.1.86:8080 ESTABLISHED 22052/filebeat $ netstat -anp |grep 8080 |grep filebeat |wc -l 624 ``` ---
**Author:** ![pmercado](https://avatars.discourse-cdn.com/v4/letter/p/59ef9b/32.png) [@pmercado](https://discuss.elastic.co/u/pmercado)\ **Post date:** [May 8, 2019, 11:55am UTC](https://discuss.elastic.co/t/kubernetes-watch-connection-never-close-when-inputreload-create-runner-failed/180112/2 "2019-05-08T11:55:02Z")
Hey @danielQ, welcome and thanks for sharing that scenario. I've trying to replicate the issue with no success. My setup is: Kubernetes 1.13, multimaster behing an `HAProxy`. Filebeat 7.0 deployed at all nodes, the configuration is retrieving logs from only one pod at the whole cluster, input reload is set to true. Then I mess up the `filebeat-inputs` configmap, adding a non valid logger. Filebeat at the log starts to complain at the reload, just as yours. I exec into the filebeat container and my connections to the `apiserver` looks good (2 established connections) Checking at the load balancer at the front of apiservers, there is no significant movement. Am I missing something to replicate your scenario? What filebeat versions are you running? ---
**Author:** ![danielQ](https://avatars.discourse-cdn.com/v4/letter/d/85f322/32.png) [@danielQ](https://discuss.elastic.co/u/danielQ)\ **Post date:** [May 9, 2019, 1:44am UTC](https://discuss.elastic.co/t/kubernetes-watch-connection-never-close-when-inputreload-create-runner-failed/180112/3 "2019-05-09T01:44:49Z")
Hi, thx for reply .filebeat 6.4.0 running in pod ,have you got the same error as me? I follower the log error and checked source code , it seems when create A newInput it will start the processors at the first line > ;[https://github.com/elastic/beats/blob/master/filebeat/input/log/input.go#L86;](https://github.com/elastic/beats/blob/master/filebeat/input/log/input.go#L86;) [https://github.com/elastic/beats/blob/master/libbeat/processors/processor.go#L84](https://github.com/elastic/beats/blob/master/libbeat/processors/processor.go#L84), and the add\_kubernets\_metadata will start watch here: > so if there any error happens in newinput process,it will create a useless watch. I'm fresh at filebeat, if I missed something please tell me, thx. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [June 6, 2019, 1:45am UTC](https://discuss.elastic.co/t/kubernetes-watch-connection-never-close-when-inputreload-create-runner-failed/180112/4 "2019-06-06T01:45:21Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.