Hi, i have installed Filebeat v7.9.1 on my Windows server and the logstash i am shipping the message is in 7.9.1 as well. However, when i ran the filebeat it seems to not get successful connection to the logstash. Previously i use Filebeat v5.1.1 and works fine sending data to Logstash v7.9.1.
FILEBEAT LOGS:
INFO instance/beat.go:640 Home path: [C:\YODA\MKZ_Filebeat_tibrvlisten_7703] Config path: [C:\YODA\MKZ_Filebeat_tibrvlisten_7703] Data path: [C:\YODA\MKZ_Filebeat_tibrvlisten_7703\data] Logs path: [C:\YODA\MKZ_Filebeat_tibrvlisten_7703\logs]
INFO instance/beat.go:648 Beat ID: 93085e9e-658e-4a26-9ee5-8d1f2ce9b469
INFO [beat] instance/beat.go:976 Beat info {"system_info": {"beat": {"path": {"config": "C:\YODA\MKZ_Filebeat_tibrvlisten_7703", "data": "C:\YODA\MKZ_Filebeat_tibrvlisten_7703\data", "home": "C:\YODA\MKZ_Filebeat_tibrvlisten_7703", "logs": "C:\YODA\MKZ_Filebeat_tibrvlisten_7703\logs"}, "type": "filebeat", "uuid": "93085e9e-658e-4a26-9ee5-8d1f2ce9b469"}}}
INFO [beat] instance/beat.go:985 Build info {"system_info": {"build": {"commit": "ad823eca4cc74439d1a44351c596c12ab51054f5", "libbeat": "7.9.1", "time": "2020-09-01T19:58:48.000Z", "version": "7.9.1"}}}
INFO [beat] instance/beat.go:988 Go runtime info {"system_info": {"go": {"os":"windows","arch":"amd64","max_procs":2,"version":"go1.14.7"}}}
INFO [beat] instance/beat.go:992 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2021-02-15T06:05:26.58+08:00","name":"MKZSA1152","ip":["172.21.0.227/24","::1/128","127.0.0.1/8"],"kernel_version":"6.3.9600.19939 (winblue_ltsb.210109-0600)","mac":["00:50:56:a3:1b:fb"],"os":{"family":"windows","platform":"windows","name":"Windows Server 2012 R2 Standard","version":"6.3","major":3,"minor":0,"patch":0,"build":"9600.19941"},"timezone":"+08","timezone_offset_sec":28800,"id":"cc3a7594-8aa2-4a5a-8544-45f2e0994a2f"}}}
INFO [beat] instance/beat.go:1021 Process info {"system_info": {"process": {"cwd": "C:\Windows\system32", "exe": "C:\YODA\MKZ_Filebeat_tibrvlisten_7703_7.9.1\filebeat.exe", "name": "filebeat.exe", "pid": 10284, "ppid": 512, "start_time": "2021-03-05T16:50:05.042+0800"}}}
INFO instance/beat.go:299 Setup Beat: filebeat; Version: 7.9.1
INFO [publisher] pipeline/module.go:113 Beat name: filebeat
WARN beater/filebeat.go:178 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
INFO instance/beat.go:450 filebeat start running.
INFO [monitoring] log/log.go:118 Starting metrics logging every 30s
INFO memlog/store.go:119 Loading data file of 'C:\YODA\MKZ_Filebeat_tibrvlisten_7703\data\registry\filebeat' succeeded. Active transaction id=57140
INFO memlog/store.go:124 Finished loading transaction log file for 'C:\YODA\MKZ_Filebeat_tibrvlisten_7703\data\registry\filebeat'. Active transaction id=69194
WARN beater/filebeat.go:381 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
INFO [registrar] registrar/registrar.go:109 States Loaded from registrar: 1
INFO [crawler] beater/crawler.go:71 Loading Inputs: 1
INFO log/input.go:157 Configured paths: [D:\YODALogs\MKZ_tibrvlisten\7703\tibrvlisten.log]
INFO [crawler] beater/crawler.go:141 Starting input (ID: 6446005666778718408)
INFO [crawler] beater/crawler.go:108 Loading and starting Inputs completed. Enabled inputs: 1
INFO cfgfile/reload.go:164 Config reloader started
INFO cfgfile/reload.go:224 Loading of config files completed.
INFO log/harvester.go:297 Harvester started for file: D:\YODALogs\MKZ_tibrvlisten\7703\tibrvlisten.log
INFO [add_cloud_metadata] add_cloud_metadata/add_cloud_metadata.go:89 add_cloud_metadata: hosting provider type not detected.
INFO [publisher_pipeline_output] pipeline/output.go:143 Connecting to backoff(async(tcp://log-app-fab.mkz-gp.icp.infineon.com:9513))
INFO [publisher] pipeline/retry.go:219 retryer: send unwait signal to consumer
INFO [publisher] pipeline/retry.go:223 done
INFO [publisher_pipeline_output] pipeline/output.go:151 Connection to backoff(async(tcp://log-app-fab.mkz-gp.icp.infineon.com:9513)) established
INFO log/harvester.go:312 File was truncated. Begin reading file from offset 0: D:\YODALogs\MKZ_tibrvlisten\7703\tibrvlisten.log
INFO log/harvester.go:297 Harvester started for file: D:\YODALogs\MKZ_tibrvlisten\7703\tibrvlisten.log
INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":515,"time":{"ms":515}},"total":{"ticks":4780,"time":{"ms":4780},"value":4780},"user":{"ticks":4265,"time":{"ms":4265}}},"handles":{"open":170},"info":{"ephemeral_id":"2c93261b-58f4-498f-b449-1e394303166b","uptime":{"ms":30139}},"memstats":{"gc_next":50640368,"memory_alloc":27153232,"memory_total":529519968,"rss":87588864},"runtime":{"goroutines":32}},"filebeat":{"events":{"active":2176,"added":43195,"done":41019},"harvester":{"closed":1,"files":{"8ecc0b49-9f5a-42e4-b23c-f47cce1518df":{"last_event_published_time":"2021-03-05T16:50:35.270Z","last_event_timestamp":"2021-03-05T16:50:35.269Z","name":"D:\YODALogs\MKZ_tibrvlisten\7703\tibrvlisten.log","read_offset":15971623,"size":3687377,"start_time":"2021-03-05T16:50:15.533Z"}},"open_files":1,"running":1,"started":2},"input":{"log":{"files":{"truncated":1}}}},"libbeat":{"config":{"module":{"running":0},"reloads":1,"scans":1},"output":{"events":{"acked":41007,"active":2048,"batches":22,"total":43055},"read":{"bytes":126},"type":"logstash","write":{"bytes":5775763}},"pipeline":{"clients":1,"events":{"active":2163,"filtered":12,"published":43170,"retry":2048,"total":43182},"queue":{"acked":41007}}},"registrar":{"states":{"current":1,"update":41019},"writes":{"success":24,"total":24}},"system":{"cpu":{"cores":2}}}}}
INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":1109,"time":{"ms":594}},"total":{"ticks":9859,"time":{"ms":5079},"value":9859},"user":{"ticks":8750,"time":{"ms":4485}}},"handles":{"open":170},"info":{"ephemeral_id":"2c93261b-58f4-498f-b449-1e394303166b","uptime":{"ms":60140}},"memstats":{"gc_next":52294256,"memory_alloc":48936256,"memory_total":1066714136,"rss":-749568},"runtime":{"goroutines":32}},"filebeat":{"events":{"active":1941,"added":51093,"done":49152},"harvester":{"files":{"8ecc0b49-9f5a-42e4-b23c-f47cce1518df":{"last_event_published_time":"2021-03-05T16:51:04.695Z","last_event_timestamp":"2021-03-05T16:51:04.695Z","read_offset":21712594,"size":28973076}},"open_files":1,"running":1}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":49152,"active":2048,"batches":25,"total":51200},"read":{"bytes":144},"write":{"bytes":6516458}},"pipeline":{"clients":1,"events":{"active":4117,"published":51105,"total":51106},"queue":{"acked":49152}}},"registrar":{"states":{"current":1,"update":49152},"writes":{"success":24,"total":24}}}}}
INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":1578,"time":{"ms":469}},"total":{"ticks":15203,"time":{"ms":5344},"value":15203},"user":{"ticks":13625,"time":{"ms":4875}}},"handles":{"open":172},"info":{"ephemeral_id":"2c93261b-58f4-498f-b449-1e394303166b","uptime":{"ms":90139}},"memstats":{"gc_next":38129056,"memory_alloc":35969008,"memory_total":1590636008,"rss":-356352},"runtime":{"goroutines":32}},"filebeat":{"events":{"added":49156,"done":49156},"harvester":{"files":{"8ecc0b49-9f5a-42e4-b23c-f47cce1518df":{"last_event_published_time":"2021-03-05T16:51:34.889Z","last_event_timestamp":"2021-03-05T16:51:34.889Z","read_offset":22434859,"size":27931887}},"open_files":1,"running":1}},"libbeat":{"config":{"module":{"running":0}},"output":{"events":{"acked":49152,"batches":24,"total":49152},"read":{"bytes":144},"write":{"bytes":6549277}},"pipeline":{"clients":1,"events":{"active":4117,"filtered":4,"published":49152,"total":49156},"queue":{"acked":49152}}},"registrar":{"states":{"current":1,"update":49156},"writes":{"success":24,"total":24}}}}}
LOGSTASH LOGS:
[INFO ][org.logstash.beats.BeatsHandler] [local: 0.0.0.0:5044, remote: 10.125.6.1:58353] Handling exception: Connection reset by peer
[WARN ][io.netty.channel.DefaultChannelPipeline] An exceptionCaught() event was fired, and it reached at the tail of the pipeline. It usually means the last handler in the pipeline did not handle the exception.
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method) ~[?:?]
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39) ~[?:?]
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:276) ~[?:?]
at sun.nio.ch.IOUtil.read(IOUtil.java:233) ~[?:?]
at sun.nio.ch.IOUtil.read(IOUtil.java:223) ~[?:?]
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:358) ~[?:?]
at io.netty.buffer.PooledByteBuf.setBytes(PooledByteBuf.java:253) ~[netty-all-4.1.49.Final.jar:4.1.49.Final]
at io.netty.buffer.AbstractByteBuf.writeBytes(AbstractByteBuf.java:1133) ~[netty-all-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.socket.nio.NioSocketChannel.doReadBytes(NioSocketChannel.java:350) ~[netty-all-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:148) ~[netty-all-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) ~[netty-all-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized(NioEventLoop.java:650) ~[netty-all-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:576) ~[netty-all-4.1.49.Final.jar:4.1.49.Final]
at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-all-4.1.49.Final.jar:4.1.49.Final]
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-all-4.1.49.Final.jar:4.1.49.Final]
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-all-4.1.49.Final.jar:4.1.49.Final]
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) [netty-all-4.1.49.Final.jar:4.1.49.Final]
at java.lang.Thread.run(Thread.java:834) [?:?]