Filebeat sending only few log entries to Logstash

I am trying to send kubernetes pod log of 100 MB to logstash via filebeat.

pod log - filebeat- logstash -syslog

I observed that filebeat stopped reading after sending some MB data. I could see the file getting created in syslog server. But I am not able to figure out the reason for it. My Logstash version is 6.6.0 and filebeat version 7.5.1. Please let me know the reason

2020-08-07T13:03:28.070Z        INFO    [monitoring]    log/log.go:145  Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":20,"time":{"ms":24}},"total":{"ticks":340,"time":{"ms":348},"value":340},"user":{"ticks":320,"time":{"ms":324}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":8},"info":{"ephemeral_id":"8ea955be-0dc4-4d9f-af81-0f073e4c0421","uptime":{"ms":30036}},"memstats":{"gc_next":27350768,"memory_alloc":13866160,"memory_total":52319368,"rss":74674176},"runtime":{"goroutines":28}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"logstash"},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":0}},"system":{"cpu":{"cores":8},"load":{"1":0.57,"15":1.09,"5":0.52,"norm":{"1":0.0712,"15":0.1363,"5":0.065}}}}}}
2020-08-07T13:03:39.477Z        WARN    [cfgwarn]       docker/input.go:49      DEPRECATED: 'docker' input deprecated. Use 'container' input instead. Will be removed in version: 8.0.0
2020-08-07T13:03:39.501Z        INFO    log/input.go:152        Configured paths: [/var/lib/docker/containers/cc160fdeea675a4352d744c0d88ee2b0eea05f9d3a09f8853f203b5036282ba7/*.log]
2020-08-07T13:03:39.501Z        INFO    input/input.go:114      Starting input of type: docker; ID: 13915751598853781751
2020-08-07T13:03:39.502Z        INFO    log/harvester.go:251    Harvester started for file: /var/lib/docker/containers/cc160fdeea675a4352d744c0d88ee2b0eea05f9d3a09f8853f203b5036282ba7/cc160fdeea675a4352d744c0d88ee2b0eea05f9d3a09f8853f203b5036282ba7-json.log
2020-08-07T13:03:39.679Z        INFO    pipeline/output.go:95   Connecting to backoff(async(tcp://10.10.127.253:5044))
2020-08-07T13:03:39.680Z        INFO    pipeline/output.go:105  Connection to backoff(async(tcp://10.10.127.253:5044)) established
2020-08-07T13:03:46.664Z        INFO    input/input.go:149      input ticker stopped
2020-08-07T13:03:46.664Z        INFO    input/input.go:167      Stopping Input: 13915751598853781751
2020-08-07T13:03:58.070Z        INFO    [monitoring]    log/log.go:145  Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":940,"time":{"ms":920}},"total":{"ticks":7550,"time":{"ms":7208},"value":7550},"user":{"ticks":6610,"time":{"ms":6288}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":9},"info":{"ephemeral_id":"8ea955be-0dc4-4d9f-af81-0f073e4c0421","uptime":{"ms":60036}},"memstats":{"gc_next":66746256,"memory_alloc":40185880,"memory_total":1190088608,"rss":61792256},"runtime":{"goroutines":29}},"filebeat":{"events":{"added":43031,"done":43031},"harvester":{"closed":1,"open_files":0,"running":0,"started":1}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":43029,"batches":22,"total":43029},"read":{"bytes":132},"write":{"bytes":3836908}},"pipeline":{"clients":0,"events":{"active":0,"filtered":2,"published":43029,"retry":2048,"total":43031},"queue":{"acked":43029}}},"registrar":{"states":{"current":18490,"update":43031},"writes":{"success":23,"total":23}},"system":{"load":{"1":1.92,"15":1.18,"5":0.86,"norm":{"1":0.24,"15":0.1475,"5":0.1075}}}}}}
2020-08-07T13:04:28.071Z        INFO    [monitoring]    log/log.go:145  Non-zero metrics in the last 30s        {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":940,"time":{"ms":4}},"total":{"ticks":7560,"time":{"ms":12},"value":7560},"user":{"ticks":6620,"time":{"ms":8}}},"handles":{"limit":{"hard":1048576,"soft":1048576},"open":9},"info":{"ephemeral_id":"8ea955be-0dc4-4d9f-af81-0f073e4c0421","uptime":{"ms":90036}},"memstats":{"gc_next":66746256,"memory_alloc":40477552,"memory_total":1190380280},"runtime":{"goroutines":29}},"filebeat":{"harvester":{"open_files":0,"running":0}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":0,"events":{"active":0}}},"registrar":{"states":{"current":18490}},"system":{"load":{"1":1.32,"15":1.16,"5":0.82,"norm":{"1":0.165,"15":0.145,"5":0.1025}}}}}}

Could you please share debug logs of Filebeat and its configuration formatted using </>?

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.