Trying to parse JSON data from file (single line JSON data), harvester is being started again and again after close_inactive time.
filebeat.yml:
filebeat.inputs:
- type: log
enabled: true
paths:
- D:\ELK\Logs\*
close_inactive: 1m
json.keys_under_root: true
json.add_error_key: true
setup.template.settings:
index.number_of_shards: 1
output.console:
pretty: true
enabled: true
JSON file:
{"error":"dial tcp 172.21.0.2:5432: getsockopt: connection refused","level":"error","msg":"Could not open database connection","time":"2017-03-25T14:57:15Z"}
Filebeat log file:
2020-02-24T15:09:31.715+0530 INFO instance/beat.go:622 Home path: [D:\ELK\Filebeat] Config path: [D:\ELK\Filebeat] Data path: [D:\ELK\Filebeat\data] Logs path: [D:\ELK\Filebeat\logs]
2020-02-24T15:09:31.817+0530 INFO instance/beat.go:630 Beat ID: e21e7115-fd1b-4878-aed5-3d8c80faae76
2020-02-24T15:09:31.820+0530 INFO [beat] instance/beat.go:958 Beat info {"system_info": {"beat": {"path": {"config": "D:\ELK\Filebeat", "data": "D:\ELK\Filebeat\data", "home": "D:\ELK\Filebeat", "logs": "D:\ELK\Filebeat\logs"}, "type": "filebeat", "uuid": "e21e7115-fd1b-4878-aed5-3d8c80faae76"}}}
2020-02-24T15:09:31.820+0530 INFO [beat] instance/beat.go:967 Build info {"system_info": {"build": {"commit": "6a23e8f8f30f5001ba344e4e54d8d9cb82cb107c", "libbeat": "7.6.0", "time": "2020-02-05T23:06:44.000Z", "version": "7.6.0"}}}
2020-02-24T15:09:31.821+0530 INFO [beat] instance/beat.go:970 Go runtime info {"system_info": {"go": {"os":"windows","arch":"amd64","max_procs":8,"version":"go1.13.7"}}}
2020-02-24T15:09:31.859+0530 INFO [beat] instance/beat.go:974 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-02-24T12:14:35.75+05:30","name":"BLL012612","ip":["fe80::5947:c7de:1965:afc/64","169.254.10.252/16","fe80::5cd9:541e:add4:5264/64","169.254.82.100/16","fe80::7979:7ee9:c7de:d796/64","169.254.215.150/16","fe80::c81e:4c6b:8f30:db38/64","10.66.27.46/22","::1/128","127.0.0.1/8"],"kernel_version":"10.0.18362.657 (WinBuild.160101.0800)","mac":["3c:2c:30:b5:9d:e2","18:1d:ea:87:af:15","1a:1d:ea:87:af:14","18:1d:ea:87:af:14"],"os":{"family":"windows","platform":"windows","name":"Windows 10 Enterprise","version":"10.0","major":10,"minor":0,"patch":0,"build":"18363.657"},"timezone":"IST","timezone_offset_sec":19800,"id":"c5e2e3f5-2c05-439b-a8a1-5285977fdd43"}}}
2020-02-24T15:09:31.860+0530 INFO [beat] instance/beat.go:1003 Process info {"system_info": {"process": {"cwd": "D:\ELK\Filebeat", "exe": "D:\ELK\Filebeat\filebeat.exe", "name": "filebeat.exe", "pid": 11696, "ppid": 16896, "start_time": "2020-02-24T15:09:30.189+0530"}}}
2020-02-24T15:09:31.860+0530 INFO instance/beat.go:298 Setup Beat: filebeat; Version: 7.6.0
2020-02-24T15:09:31.860+0530 INFO [publisher] pipeline/module.go:110 Beat name: BLL012612
2020-02-24T15:09:31.866+0530 WARN beater/filebeat.go:152 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2020-02-24T15:09:31.866+0530 INFO instance/beat.go:439 filebeat start running.
2020-02-24T15:09:31.866+0530 INFO [monitoring] log/log.go:118 Starting metrics logging every 30s
2020-02-24T15:09:31.870+0530 INFO registrar/registrar.go:145 Loading registrar data from D:\ELK\Filebeat\data\registry\filebeat\data.json
2020-02-24T15:09:31.871+0530 INFO registrar/registrar.go:152 States Loaded from registrar: 1
2020-02-24T15:09:31.871+0530 WARN beater/filebeat.go:368 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2020-02-24T15:09:31.871+0530 INFO crawler/crawler.go:72 Loading Inputs: 1
2020-02-24T15:09:31.872+0530 INFO log/input.go:152 Configured paths: [D:\ELK\Logs*]
2020-02-24T15:09:31.872+0530 INFO input/input.go:114 Starting input of type: log; ID: 5965211091724442982
2020-02-24T15:09:31.872+0530 INFO crawler/crawler.go:106 Loading and starting Inputs completed. Enabled inputs: 1
2020-02-24T15:09:31.875+0530 INFO log/harvester.go:297 Harvester started for file: D:\ELK\Logs\new
2020-02-24T15:10:01.872+0530 INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":281,"time":{"ms":281}},"total":{"ticks":468,"time":{"ms":468},"value":468},"user":{"ticks":187,"time":{"ms":187}}},"handles":{"open":320},"info":{"ephemeral_id":"3dfb7be8-cfc8-45df-a9f9-bb5e1199f33a","uptime":{"ms":30494}},"memstats":{"gc_next":8238688,"memory_alloc":4838568,"memory_total":12987616,"rss":38322176},"runtime":{"goroutines":27}},"filebeat":{"events":{"added":2,"done":2},"harvester":{"files":{"098541cf-b46c-47a9-8793-c00fd23f0165":{"last_event_published_time":"","last_event_timestamp":"","name":"D:\ELK\Logs\new","size":157,"start_time":"2020-02-24T15:09:31.875Z"}},"open_files":1,"running":1,"started":1}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"console"},"pipeline":{"clients":1,"events":{"active":0,"filtered":2,"total":2}}},"registrar":{"states":{"current":1,"update":2},"writes":{"success":2,"total":2}},"system":{"cpu":{"cores":8}}}}}
2020-02-24T15:10:31.871+0530 INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":312,"time":{"ms":31}},"total":{"ticks":515,"time":{"ms":47},"value":515},"user":{"ticks":203,"time":{"ms":16}}},"handles":{"open":314},"info":{"ephemeral_id":"3dfb7be8-cfc8-45df-a9f9-bb5e1199f33a","uptime":{"ms":60494}},"memstats":{"gc_next":8238688,"memory_alloc":4917792,"memory_total":13066840,"rss":65536},"runtime":{"goroutines":27}},"filebeat":{"harvester":{"open_files":1,"running":1}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":1}}}}}
2020-02-24T15:10:36.902+0530 INFO log/harvester.go:324 File is inactive: D:\ELK\Logs\new. Closing because close_inactive of 1m0s reached.
2020-02-24T15:10:41.918+0530 INFO log/harvester.go:297 Harvester started for file: D:\ELK\Logs\new
2020-02-24T15:11:01.868+0530 INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":312},"total":{"ticks":515,"value":515},"user":{"ticks":203}},"handles":{"open":316},"info":{"ephemeral_id":"3dfb7be8-cfc8-45df-a9f9-bb5e1199f33a","uptime":{"ms":90491}},"memstats":{"gc_next":8238688,"memory_alloc":5045128,"memory_total":13194176,"rss":49152},"runtime":{"goroutines":27}},"filebeat":{"events":{"added":2,"done":2},"harvester":{"closed":1,"files":{"6e5bb668-7a8f-41d1-aa05-712bf26374bc":{"last_event_published_time":"","last_event_timestamp":"","name":"D:\ELK\Logs\new","size":157,"start_time":"2020-02-24T15:10:41.917Z"}},"open_files":1,"running":1,"started":1}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0,"filtered":2,"total":2}}},"registrar":{"states":{"current":1,"update":2},"writes":{"success":2,"total":2}}}}}
2020-02-24T15:11:31.872+0530 INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":312},"total":{"ticks":530,"time":{"ms":15},"value":530},"user":{"ticks":218,"time":{"ms":15}}},"handles":{"open":322},"info":{"ephemeral_id":"3dfb7be8-cfc8-45df-a9f9-bb5e1199f33a","uptime":{"ms":120497}},"memstats":{"gc_next":9011760,"memory_alloc":4992976,"memory_total":13256024,"rss":163840},"runtime":{"goroutines":27}},"filebeat":{"harvester":{"open_files":1,"running":1}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":1}}}}}
2020-02-24T15:11:46.944+0530 INFO log/harvester.go:324 File is inactive: D:\ELK\Logs\new. Closing because close_inactive of 1m0s reached.
2020-02-24T15:11:51.955+0530 INFO log/harvester.go:297 Harvester started for file: D:\ELK\Logs\new
2020-02-24T15:12:01.872+0530 INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":343,"time":{"ms":31}},"total":{"ticks":561,"time":{"ms":31},"value":561},"user":{"ticks":218}},"handles":{"open":320},"info":{"ephemeral_id":"3dfb7be8-cfc8-45df-a9f9-bb5e1199f33a","uptime":{"ms":150496}},"memstats":{"gc_next":9011760,"memory_alloc":4638128,"memory_total":13392112,"rss":-700416},"runtime":{"goroutines":27}},"filebeat":{"events":{"added":2,"done":2},"harvester":{"closed":1,"files":{"d3b1057e-c27f-4121-bc75-38278c5016ab":{"last_event_published_time":"","last_event_timestamp":"","name":"D:\ELK\Logs\new","size":157,"start_time":"2020-02-24T15:11:51.954Z"}},"open_files":1,"running":1,"started":1}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0,"filtered":2,"total":2}}},"registrar":{"states":{"current":1,"update":2},"writes":{"success":2,"total":2}}}}}
No output on console