# Harvester in loop while parsing JSON data
**URL:** https://discuss.elastic.co/t/harvester-in-loop-while-parsing-json-data/220612
**Category:** Beats
**Tags:** filebeat
**Created:** [February 24, 2020, 9:43am UTC](https://discuss.elastic.co/t/harvester-in-loop-while-parsing-json-data/220612 "2020-02-24T09:43:30Z")
**Posts on this page:** 8
**Page:** 1
### Author:  [@aksroh](https://discuss.elastic.co/u/aksroh)
#### Post date: [February 24, 2020, 9:43am UTC](https://discuss.elastic.co/t/harvester-in-loop-while-parsing-json-data/220612/1 "2020-02-24T09:43:31Z")
Trying to parse JSON data from file (single line JSON data), harvester is being started again and again after close\_inactive time.
filebeat.yml:
```
filebeat.inputs:
- type: log
enabled: true
paths:
- D:\ELK\Logs\*
close_inactive: 1m
json.keys_under_root: true
json.add_error_key: true
setup.template.settings:
index.number_of_shards: 1
output.console:
pretty: true
enabled: true
```
JSON file:
`{"error":"dial tcp 172.21.0.2:5432: getsockopt: connection refused","level":"error","msg":"Could not open database connection","time":"2017-03-25T14:57:15Z"}`
Filebeat log file:
2020-02-24T15:09:31.715+0530 INFO instance/beat.go:622 Home path: [D:\ELK\Filebeat] Config path: [D:\ELK\Filebeat] Data path: [D:\ELK\Filebeat\data] Logs path: [D:\ELK\Filebeat\logs]
2020-02-24T15:09:31.817+0530 INFO instance/beat.go:630 Beat ID: e21e7115-fd1b-4878-aed5-3d8c80faae76
2020-02-24T15:09:31.820+0530 INFO [beat] instance/beat.go:958 Beat info {"system\_info": {"beat": {"path": {"config": "D:\ELK\Filebeat", "data": "D:\ELK\Filebeat\data", "home": "D:\ELK\Filebeat", "logs": "D:\ELK\Filebeat\logs"}, "type": "filebeat", "uuid": "e21e7115-fd1b-4878-aed5-3d8c80faae76"}}}
2020-02-24T15:09:31.820+0530 INFO [beat] instance/beat.go:967 Build info {"system\_info": {"build": {"commit": "6a23e8f8f30f5001ba344e4e54d8d9cb82cb107c", "libbeat": "7.6.0", "time": "2020-02-05T23:06:44.000Z", "version": "7.6.0"}}}
2020-02-24T15:09:31.821+0530 INFO [beat] instance/beat.go:970 Go runtime info {"system\_info": {"go": {"os":"windows","arch":"amd64","max\_procs":8,"version":"go1.13.7"}}}
2020-02-24T15:09:31.859+0530 INFO [beat] instance/beat.go:974 Host info {"system\_info": {"host": {"architecture":"x86\_64","boot\_time":"2020-02-24T12:14:35.75+05:30","name":"BLL012612","ip":["fe80::5947:c7de:1965:afc/64","169.254.10.252/16","fe80::5cd9:541e:add4:5264/64","169.254.82.100/16","fe80::7979:7ee9:c7de:d796/64","169.254.215.150/16","fe80::c81e:4c6b:8f30:db38/64","10.66.27.46/22","::1/128","127.0.0.1/8"],"kernel\_version":"10.0.18362.657 (WinBuild.160101.0800)","mac":["3c:2c:30:b5:9d:e2","18:1d:ea:87:af:15","1a:1d:ea:87:af:14","18:1d:ea:87:af:14"],"os":{"family":"windows","platform":"windows","name":"Windows 10 Enterprise","version":"10.0","major":10,"minor":0,"patch":0,"build":"18363.657"},"timezone":"IST","timezone\_offset\_sec":19800,"id":"c5e2e3f5-2c05-439b-a8a1-5285977fdd43"}}}
2020-02-24T15:09:31.860+0530 INFO [beat] instance/beat.go:1003 Process info {"system\_info": {"process": {"cwd": "D:\ELK\Filebeat", "exe": "D:\ELK\Filebeat\filebeat.exe", "name": "filebeat.exe", "pid": 11696, "ppid": 16896, "start\_time": "2020-02-24T15:09:30.189+0530"}}}
2020-02-24T15:09:31.860+0530 INFO instance/beat.go:298 Setup Beat: filebeat; Version: 7.6.0
2020-02-24T15:09:31.860+0530 INFO [publisher] pipeline/module.go:110 Beat name: BLL012612
2020-02-24T15:09:31.866+0530 WARN beater/filebeat.go:152 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2020-02-24T15:09:31.866+0530 INFO instance/beat.go:439 filebeat start running.
2020-02-24T15:09:31.866+0530 INFO [monitoring] log/log.go:118 Starting metrics logging every 30s
2020-02-24T15:09:31.870+0530 INFO registrar/registrar.go:145 Loading registrar data from D:\ELK\Filebeat\data\registry\filebeat\data.json
2020-02-24T15:09:31.871+0530 INFO registrar/registrar.go:152 States Loaded from registrar: 1
2020-02-24T15:09:31.871+0530 WARN beater/filebeat.go:368 Filebeat is unable to load the Ingest Node pipelines for the configured modules because the Elasticsearch output is not configured/enabled. If you have already loaded the Ingest Node pipelines or are using Logstash pipelines, you can ignore this warning.
2020-02-24T15:09:31.871+0530 INFO crawler/crawler.go:72 Loading Inputs: 1
2020-02-24T15:09:31.872+0530 INFO log/input.go:152 Configured paths: [D:\ELK\Logs\*]
2020-02-24T15:09:31.872+0530 INFO input/input.go:114 Starting input of type: log; ID: 5965211091724442982
2020-02-24T15:09:31.872+0530 INFO crawler/crawler.go:106 Loading and starting Inputs completed. Enabled inputs: 1
2020-02-24T15:09:31.875+0530 INFO log/harvester.go:297 Harvester started for file: D:\ELK\Logs\new
2020-02-24T15:10:01.872+0530 INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":281,"time":{"ms":281}},"total":{"ticks":468,"time":{"ms":468},"value":468},"user":{"ticks":187,"time":{"ms":187}}},"handles":{"open":320},"info":{"ephemeral\_id":"3dfb7be8-cfc8-45df-a9f9-bb5e1199f33a","uptime":{"ms":30494}},"memstats":{"gc\_next":8238688,"memory\_alloc":4838568,"memory\_total":12987616,"rss":38322176},"runtime":{"goroutines":27}},"filebeat":{"events":{"added":2,"done":2},"harvester":{"files":{"098541cf-b46c-47a9-8793-c00fd23f0165":{"last\_event\_published\_time":"","last\_event\_timestamp":"","name":"D:\ELK\Logs\new","size":157,"start\_time":"2020-02-24T15:09:31.875Z"}},"open\_files":1,"running":1,"started":1}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"console"},"pipeline":{"clients":1,"events":{"active":0,"filtered":2,"total":2}}},"registrar":{"states":{"current":1,"update":2},"writes":{"success":2,"total":2}},"system":{"cpu":{"cores":8}}}}}
2020-02-24T15:10:31.871+0530 INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":312,"time":{"ms":31}},"total":{"ticks":515,"time":{"ms":47},"value":515},"user":{"ticks":203,"time":{"ms":16}}},"handles":{"open":314},"info":{"ephemeral\_id":"3dfb7be8-cfc8-45df-a9f9-bb5e1199f33a","uptime":{"ms":60494}},"memstats":{"gc\_next":8238688,"memory\_alloc":4917792,"memory\_total":13066840,"rss":65536},"runtime":{"goroutines":27}},"filebeat":{"harvester":{"open\_files":1,"running":1}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":1}}}}}
2020-02-24T15:10:36.902+0530 INFO log/harvester.go:324 File is inactive: D:\ELK\Logs\new. Closing because close\_inactive of 1m0s reached.
2020-02-24T15:10:41.918+0530 INFO log/harvester.go:297 Harvester started for file: D:\ELK\Logs\new
2020-02-24T15:11:01.868+0530 INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":312},"total":{"ticks":515,"value":515},"user":{"ticks":203}},"handles":{"open":316},"info":{"ephemeral\_id":"3dfb7be8-cfc8-45df-a9f9-bb5e1199f33a","uptime":{"ms":90491}},"memstats":{"gc\_next":8238688,"memory\_alloc":5045128,"memory\_total":13194176,"rss":49152},"runtime":{"goroutines":27}},"filebeat":{"events":{"added":2,"done":2},"harvester":{"closed":1,"files":{"6e5bb668-7a8f-41d1-aa05-712bf26374bc":{"last\_event\_published\_time":"","last\_event\_timestamp":"","name":"D:\ELK\Logs\new","size":157,"start\_time":"2020-02-24T15:10:41.917Z"}},"open\_files":1,"running":1,"started":1}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0,"filtered":2,"total":2}}},"registrar":{"states":{"current":1,"update":2},"writes":{"success":2,"total":2}}}}}
2020-02-24T15:11:31.872+0530 INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":312},"total":{"ticks":530,"time":{"ms":15},"value":530},"user":{"ticks":218,"time":{"ms":15}}},"handles":{"open":322},"info":{"ephemeral\_id":"3dfb7be8-cfc8-45df-a9f9-bb5e1199f33a","uptime":{"ms":120497}},"memstats":{"gc\_next":9011760,"memory\_alloc":4992976,"memory\_total":13256024,"rss":163840},"runtime":{"goroutines":27}},"filebeat":{"harvester":{"open\_files":1,"running":1}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":1}}}}}
2020-02-24T15:11:46.944+0530 INFO log/harvester.go:324 File is inactive: D:\ELK\Logs\new. Closing because close\_inactive of 1m0s reached.
2020-02-24T15:11:51.955+0530 INFO log/harvester.go:297 Harvester started for file: D:\ELK\Logs\new
2020-02-24T15:12:01.872+0530 INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":343,"time":{"ms":31}},"total":{"ticks":561,"time":{"ms":31},"value":561},"user":{"ticks":218}},"handles":{"open":320},"info":{"ephemeral\_id":"3dfb7be8-cfc8-45df-a9f9-bb5e1199f33a","uptime":{"ms":150496}},"memstats":{"gc\_next":9011760,"memory\_alloc":4638128,"memory\_total":13392112,"rss":-700416},"runtime":{"goroutines":27}},"filebeat":{"events":{"added":2,"done":2},"harvester":{"closed":1,"files":{"d3b1057e-c27f-4121-bc75-38278c5016ab":{"last\_event\_published\_time":"","last\_event\_timestamp":"","name":"D:\ELK\Logs\new","size":157,"start\_time":"2020-02-24T15:11:51.954Z"}},"open\_files":1,"running":1,"started":1}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0,"filtered":2,"total":2}}},"registrar":{"states":{"current":1,"update":2},"writes":{"success":2,"total":2}}}}}
No output on console
---
### Author:  [@aksroh](https://discuss.elastic.co/u/aksroh)
#### Post date: [February 24, 2020, 10:21am UTC](https://discuss.elastic.co/t/harvester-in-loop-while-parsing-json-data/220612/3 "2020-02-24T10:21:35Z")
Tried increasing `close_inactive` time and used `decode_json_fields`, still harvester in loop.
filebeat.yml:
```
filebeat.inputs:
- type: log
enabled: true
paths:
- D:\ELK\Logs\*
close_inactive: 10m
setup.template.settings:
index.number_of_shards: 1
processors:
- decode_json_fields:
fields: ["message"]
process_array: true
max_depth: 5
overwrite_keys: true
output.console:
pretty: true
enabled: true
```
filebeat log file:
```
2020-02-24T15:29:04.052+0530 INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":531},"total":{"ticks":796,"value":796},"user":{"ticks":265}},"handles":{"open":337},"info":{"ephemeral_id":"9b2772a0-add8-420f-9401-2aad018977af","uptime":{"ms":570565}},"memstats":{"gc_next":8925024,"memory_alloc":4605448,"memory_total":14121576,"rss":4096},"runtime":{"goroutines":27}},"filebeat":{"harvester":{"open_files":1,"running":1}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":1}}}}}
2020-02-24T15:29:34.049+0530 INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":531},"total":{"ticks":796,"value":796},"user":{"ticks":265}},"handles":{"open":337},"info":{"ephemeral_id":"9b2772a0-add8-420f-9401-2aad018977af","uptime":{"ms":600564}},"memstats":{"gc_next":8925024,"memory_alloc":4668320,"memory_total":14184448,"rss":53248},"runtime":{"goroutines":27}},"filebeat":{"harvester":{"open_files":1,"running":1}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":1}}}}}
2020-02-24T15:29:39.152+0530 INFO log/harvester.go:324 File is inactive: D:\ELK\Logs\new. Closing because close_inactive of 10m0s reached.
2020-02-24T15:29:44.277+0530 INFO log/harvester.go:297 Harvester started for file: D:\ELK\Logs\new
2020-02-24T15:30:04.050+0530 INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":531},"total":{"ticks":796,"value":796},"user":{"ticks":265}},"handles":{"open":339},"info":{"ephemeral_id":"9b2772a0-add8-420f-9401-2aad018977af","uptime":{"ms":630565}},"memstats":{"gc_next":8926416,"memory_alloc":4549864,"memory_total":14304640,"rss":98304},"runtime":{"goroutines":27}},"filebeat":{"events":{"added":2,"done":2},"harvester":{"closed":1,"files":{"cf385e6d-8d2e-4bbf-a885-cd077f23688d":{"last_event_published_time":"","last_event_timestamp":"","name":"D:\\ELK\\Logs\\new","size":157,"start_time":"2020-02-24T15:29:44.276Z"}},"open_files":1,"running":1,"started":1}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0,"filtered":2,"total":2}}},"registrar":{"states":{"current":1,"update":2},"writes":{"success":2,"total":2}}}}}
2020-02-24T15:30:34.049+0530 INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":531},"total":{"ticks":796,"value":796},"user":{"ticks":265}},"handles":{"open":339},"info":{"ephemeral_id":"9b2772a0-add8-420f-9401-2aad018977af","uptime":{"ms":660564}},"memstats":{"gc_next":8926416,"memory_alloc":4616432,"memory_total":14371208},"runtime":{"goroutines":27}},"filebeat":{"harvester":{"open_files":1,"running":1}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":1}}}}}
2020-02-24T15:31:04.050+0530 INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":531},"total":{"ticks":796,"value":796},"user":{"ticks":265}},"handles":{"open":337},"info":{"ephemeral_id":"9b2772a0-add8-420f-9401-2aad018977af","uptime":{"ms":690564}},"memstats":{"gc_next":8926416,"memory_alloc":4675752,"memory_total":14430528},"runtime":{"goroutines":27}},"filebeat":{"harvester":{"open_files":1,"running":1}},"libbeat":{"config":{"module":{"running":0}},"pipeline":{"clients":1,"events":{"active":0}}},"registrar":{"states":{"current":1}}}}}
```
---
### Author:  [@ChrsMark](https://discuss.elastic.co/u/ChrsMark)
#### Post date: [February 24, 2020, 11:54am UTC](https://discuss.elastic.co/t/harvester-in-loop-while-parsing-json-data/220612/4 "2020-02-24T11:54:33Z")
Hi!
If you remove completely all json decoding related configuration, are the logs collected (with no parsing of course)?
C.
---
### Author:  [@aksroh](https://discuss.elastic.co/u/aksroh)
#### Post date: [February 24, 2020, 12:22pm UTC](https://discuss.elastic.co/t/harvester-in-loop-while-parsing-json-data/220612/5 "2020-02-24T12:22:40Z")
@ChrsMark no the logs are not collected
filebeat.yml
```
filebeat.inputs:
- type: log
enabled: true
paths:
- D:\ELK\Logs\*
setup.template.settings:
index.number_of_shards: 1
output.elasticsearch:
hosts: ["localhost:9200"]
logging.level: debug
```
filebeat log file:
```
2020-02-24T17:50:42.212+0530 INFO instance/beat.go:622 Home path: [D:\ELK\Filebeat] Config path: [D:\ELK\Filebeat] Data path: [D:\ELK\Filebeat\data] Logs path: [D:\ELK\Filebeat\logs]
2020-02-24T17:50:42.216+0530 DEBUG [beat] instance/beat.go:674 Beat metadata path: D:\ELK\Filebeat\data\meta.json
2020-02-24T17:50:42.306+0530 INFO instance/beat.go:630 Beat ID: e3605419-2e02-4f97-811a-e9d1cb4cf622
2020-02-24T17:50:42.308+0530 DEBUG [seccomp] seccomp/seccomp.go:96 Syscall filtering is only supported on Linux
2020-02-24T17:50:42.308+0530 INFO [beat] instance/beat.go:958 Beat info {"system_info": {"beat": {"path": {"config": "D:\\ELK\\Filebeat", "data": "D:\\ELK\\Filebeat\\data", "home": "D:\\ELK\\Filebeat", "logs": "D:\\ELK\\Filebeat\\logs"}, "type": "filebeat", "uuid": "e3605419-2e02-4f97-811a-e9d1cb4cf622"}}}
2020-02-24T17:50:42.308+0530 INFO [beat] instance/beat.go:967 Build info {"system_info": {"build": {"commit": "6a23e8f8f30f5001ba344e4e54d8d9cb82cb107c", "libbeat": "7.6.0", "time": "2020-02-05T23:06:44.000Z", "version": "7.6.0"}}}
2020-02-24T17:50:42.308+0530 INFO [beat] instance/beat.go:970 Go runtime info {"system_info": {"go": {"os":"windows","arch":"amd64","max_procs":8,"version":"go1.13.7"}}}
2020-02-24T17:50:42.343+0530 INFO [beat] instance/beat.go:974 Host info {"system_info": {"host": {"architecture":"x86_64","boot_time":"2020-02-24T12:14:35.74+05:30","name":"BLL012612","ip":["fe80::5947:c7de:1965:afc/64","169.254.10.252/16","fe80::5cd9:541e:add4:5264/64","169.254.82.100/16","fe80::7979:7ee9:c7de:d796/64","169.254.215.150/16","fe80::c81e:4c6b:8f30:db38/64","10.66.27.46/22","::1/128","127.0.0.1/8"],"kernel_version":"10.0.18362.657 (WinBuild.160101.0800)","mac":["3c:2c:30:b5:9d:e2","18:1d:ea:87:af:15","1a:1d:ea:87:af:14","18:1d:ea:87:af:14"],"os":{"family":"windows","platform":"windows","name":"Windows 10 Enterprise","version":"10.0","major":10,"minor":0,"patch":0,"build":"18363.657"},"timezone":"IST","timezone_offset_sec":19800,"id":"c5e2e3f5-2c05-439b-a8a1-5285977fdd43"}}}
2020-02-24T17:50:42.343+0530 INFO [beat] instance/beat.go:1003 Process info {"system_info": {"process": {"cwd": "D:\\ELK\\Filebeat", "exe": "D:\\ELK\\Filebeat\\filebeat.exe", "name": "filebeat.exe", "pid": 17368, "ppid": 15952, "start_time": "2020-02-24T17:50:40.348+0530"}}}
2020-02-24T17:50:42.343+0530 INFO instance/beat.go:298 Setup Beat: filebeat; Version: 7.6.0
2020-02-24T17:50:42.343+0530 DEBUG [beat] instance/beat.go:324 Initializing output plugins
2020-02-24T17:50:42.343+0530 INFO [index-management] idxmgmt/std.go:182 Set output.elasticsearch.index to 'filebeat-7.6.0' as ILM is enabled.
2020-02-24T17:50:42.344+0530 INFO elasticsearch/client.go:174 Elasticsearch url: http://localhost:9200
2020-02-24T17:50:42.344+0530 DEBUG [publisher] pipeline/consumer.go:137 start pipeline event consumer
2020-02-24T17:50:42.344+0530 INFO [publisher] pipeline/module.go:110 Beat name: BLL012612
2020-02-24T17:50:42.347+0530 INFO instance/beat.go:439 filebeat start running.
2020-02-24T17:50:42.347+0530 INFO [monitoring] log/log.go:118 Starting metrics logging every 30s
2020-02-24T17:50:42.347+0530 DEBUG [test] registrar/migrate.go:159 isFile(D:\ELK\Filebeat\data\registry) -> false
2020-02-24T17:50:42.347+0530 DEBUG [service] service/service_windows.go:72 Windows is interactive: true
2020-02-24T17:50:42.347+0530 DEBUG [test] registrar/migrate.go:159 isFile() -> false
2020-02-24T17:50:42.347+0530 DEBUG [test] registrar/migrate.go:152 isDir(D:\ELK\Filebeat\data\registry\filebeat) -> false
2020-02-24T17:50:42.347+0530 DEBUG [registrar] registrar/migrate.go:51 Registry type '' found
2020-02-24T17:50:42.347+0530 DEBUG [test] registrar/migrate.go:159 isFile(.bak) -> false
2020-02-24T17:50:42.347+0530 DEBUG [test] registrar/migrate.go:152 isDir(D:\ELK\Filebeat\data\registry\filebeat) -> false
2020-02-24T17:50:42.347+0530 INFO registrar/migrate.go:104 No registry home found. Create: D:\ELK\Filebeat\data\registry\filebeat
2020-02-24T17:50:42.348+0530 DEBUG [test] registrar/migrate.go:159 isFile(D:\ELK\Filebeat\data\registry\filebeat\meta.json) -> false
2020-02-24T17:50:42.348+0530 INFO registrar/migrate.go:112 Initialize registry meta file
2020-02-24T17:50:42.384+0530 INFO registrar/registrar.go:108 No registry file found under: D:\ELK\Filebeat\data\registry\filebeat\data.json. Creating a new registry file.
2020-02-24T17:50:42.384+0530 DEBUG [registrar] registrar/registrar.go:411 Write registry file: D:\ELK\Filebeat\data\registry\filebeat\data.json (0)
2020-02-24T17:50:42.424+0530 DEBUG [registrar] registrar/registrar.go:404 Registry file updated. 0 states written.
2020-02-24T17:50:42.427+0530 INFO registrar/registrar.go:145 Loading registrar data from D:\ELK\Filebeat\data\registry\filebeat\data.json
2020-02-24T17:50:42.427+0530 INFO registrar/registrar.go:152 States Loaded from registrar: 0
2020-02-24T17:50:42.427+0530 INFO crawler/crawler.go:72 Loading Inputs: 1
2020-02-24T17:50:42.427+0530 DEBUG [registrar] registrar/registrar.go:278 Starting Registrar
2020-02-24T17:50:42.428+0530 DEBUG [input] log/config.go:204 recursive glob enabled
2020-02-24T17:50:42.428+0530 DEBUG [input] log/input.go:164 exclude_files: []. Number of stats: 0
2020-02-24T17:50:42.428+0530 DEBUG [input] log/input.go:185 input with previous states loaded: 0
2020-02-24T17:50:42.428+0530 INFO log/input.go:152 Configured paths: [D:\ELK\Logs\*]
2020-02-24T17:50:42.428+0530 INFO input/input.go:114 Starting input of type: log; ID: 4960765251822949949
2020-02-24T17:50:42.428+0530 INFO crawler/crawler.go:106 Loading and starting Inputs completed. Enabled inputs: 1
2020-02-24T17:50:42.428+0530 DEBUG [input] log/input.go:191 Start next scan
2020-02-24T17:50:42.430+0530 DEBUG [input] log/input.go:421 Check file for harvesting: D:\ELK\Logs\new
2020-02-24T17:50:42.430+0530 DEBUG [input] log/input.go:494 Start harvester for new file: D:\ELK\Logs\new
2020-02-24T17:50:42.433+0530 DEBUG [harvester] log/harvester.go:573 Setting offset for file based on seek: D:\ELK\Logs\new
2020-02-24T17:50:42.434+0530 DEBUG [harvester] log/harvester.go:559 Setting offset for file: D:\ELK\Logs\new. Offset: 0
2020-02-24T17:50:42.434+0530 DEBUG [harvester] log/harvester.go:205 Harvester setup successful. Line terminator: 1
2020-02-24T17:50:42.434+0530 DEBUG [acker] beater/acker.go:64 stateful ack {"count": 1}
2020-02-24T17:50:42.434+0530 DEBUG [publisher] pipeline/client.go:220 Pipeline client receives callback 'onFilteredOut' for event: {Timestamp:0001-01-01 00:00:00 +0000 UTC Meta:null Fields:null Private:{Id: Finished:false Fileinfo:0xc0001b9110 Source:D:\ELK\Logs\new Offset:0 Timestamp:2020-02-24 17:50:42.4305129 +0530 IST m=+0.509084101 TTL:-1ns Type:log Meta:map[] FileStateOS:458752-111934-3233102111} TimeSeries:false}
2020-02-24T17:50:42.434+0530 DEBUG [registrar] registrar/registrar.go:356 Processing 1 events
2020-02-24T17:50:42.434+0530 DEBUG [harvester] log/harvester.go:478 Update state: D:\ELK\Logs\new, offset: 0
2020-02-24T17:50:42.434+0530 DEBUG [input] file/states.go:68 New state added for D:\ELK\Logs\new
2020-02-24T17:50:42.434+0530 DEBUG [input] file/states.go:68 New state added for D:\ELK\Logs\new
2020-02-24T17:50:42.434+0530 DEBUG [registrar] registrar/registrar.go:326 Registrar state updates processed. Count: 1
2020-02-24T17:50:42.434+0530 DEBUG [registrar] registrar/registrar.go:411 Write registry file: D:\ELK\Filebeat\data\registry\filebeat\data.json (1)
2020-02-24T17:50:42.434+0530 INFO log/harvester.go:297 Harvester started for file: D:\ELK\Logs\new
2020-02-24T17:50:42.434+0530 DEBUG [input] log/input.go:212 input states cleaned up. Before: 1, After: 1, Pending: 0
2020-02-24T17:50:42.434+0530 DEBUG [harvester] log/log.go:107 End of file reached: D:\ELK\Logs\new; Backoff now.
2020-02-24T17:50:42.473+0530 DEBUG [registrar] registrar/registrar.go:404 Registry file updated. 1 states written.
2020-02-24T17:50:43.446+0530 DEBUG [harvester] log/log.go:107 End of file reached: D:\ELK\Logs\new; Backoff now.
2020-02-24T17:50:45.453+0530 DEBUG [harvester] log/log.go:107 End of file reached: D:\ELK\Logs\new; Backoff now.
2020-02-24T17:50:49.454+0530 DEBUG [harvester] log/log.go:107 End of file reached: D:\ELK\Logs\new; Backoff now.
2020-02-24T17:50:52.435+0530 DEBUG [input] input/input.go:152 Run input
2020-02-24T17:50:52.435+0530 DEBUG [input] log/input.go:191 Start next scan
2020-02-24T17:50:52.438+0530 DEBUG [input] log/input.go:421 Check file for harvesting: D:\ELK\Logs\new
2020-02-24T17:50:52.438+0530 DEBUG [input] log/input.go:511 Update existing file for harvesting: D:\ELK\Logs\new, offset: 0
2020-02-24T17:50:52.438+0530 DEBUG [input] log/input.go:563 Harvester for file is still running: D:\ELK\Logs\new
2020-02-24T17:50:52.438+0530 DEBUG [input] log/input.go:212 input states cleaned up. Before: 1, After: 1, Pending: 0
2020-02-24T17:50:57.456+0530 DEBUG [harvester] log/log.go:107 End of file reached: D:\ELK\Logs\new; Backoff now.
2020-02-24T17:51:02.440+0530 DEBUG [input] input/input.go:152 Run input
2020-02-24T17:51:02.440+0530 DEBUG [input] log/input.go:191 Start next scan
2020-02-24T17:51:02.443+0530 DEBUG [input] log/input.go:421 Check file for harvesting: D:\ELK\Logs\new
2020-02-24T17:51:02.443+0530 DEBUG [input] log/input.go:511 Update existing file for harvesting: D:\ELK\Logs\new, offset: 0
2020-02-24T17:51:02.443+0530 DEBUG [input] log/input.go:563 Harvester for file is still running: D:\ELK\Logs\new
2020-02-24T17:51:02.443+0530 DEBUG [input] log/input.go:212 input states cleaned up. Before: 1, After: 1, Pending: 0
2020-02-24T17:51:07.461+0530 DEBUG [harvester] log/log.go:107 End of file reached: D:\ELK\Logs\new; Backoff now.
2020-02-24T17:51:12.347+0530 INFO [monitoring] log/log.go:145 Non-zero metrics in the last 30s {"monitoring": {"metrics": {"beat":{"cpu":{"system":{"ticks":62,"time":{"ms":62}},"total":{"ticks":93,"time":{"ms":93},"value":0},"user":{"ticks":31,"time":{"ms":31}}},"handles":{"open":318},"info":{"ephemeral_id":"9a36fd6d-ef18-49e6-8f12-096427d7cc05","uptime":{"ms":30401}},"memstats":{"gc_next":8451104,"memory_alloc":4672216,"memory_total":13041192,"rss":39067648},"runtime":{"goroutines":27}},"filebeat":{"events":{"added":1,"done":1},"harvester":{"files":{"da8e9c12-0975-4195-bd02-e3f1990490e4":{"last_event_published_time":"","last_event_timestamp":"","name":"D:\\ELK\\Logs\\new","size":42,"start_time":"2020-02-24T17:50:42.434Z"}},"open_files":1,"running":1,"started":1}},"libbeat":{"config":{"module":{"running":0}},"output":{"type":"elasticsearch"},"pipeline":{"clients":1,"events":{"active":0,"filtered":1,"total":1}}},"registrar":{"states":{"current":1,"update":1},"writes":{"success":2,"total":2}},"system":{"cpu":{"cores":8}}}}}
2020-02-24T17:51:12.444+0530 DEBUG [input] input/input.go:152 Run input
2020-02-24T17:51:12.444+0530 DEBUG [input] log/input.go:191 Start next scan
2020-02-24T17:51:12.445+0530 DEBUG [input] log/input.go:421 Check file for harvesting: D:\ELK\Logs\new
2020-02-24T17:51:12.445+0530 DEBUG [input] log/input.go:511 Update existing file for harvesting: D:\ELK\Logs\new, offset: 0
2020-02-24T17:51:12.445+0530 DEBUG [input] log/input.go:563 Harvester for file is still running: D:\ELK\Logs\new
2020-02-24T17:51:12.445+0530 DEBUG [input] log/input.go:212 input states cleaned up. Before: 1, After: 1, Pending: 0
```
---
### Author:  [@ChrsMark](https://discuss.elastic.co/u/ChrsMark)
#### Post date: [February 24, 2020, 2:03pm UTC](https://discuss.elastic.co/t/harvester-in-loop-while-parsing-json-data/220612/6 "2020-02-24T14:03:41Z")
Are new logs being created?
Could you reset the registries by removing the `data` folder?
C.
---
### Author:  [@aksroh](https://discuss.elastic.co/u/aksroh)
#### Post date: [February 25, 2020, 4:39am UTC](https://discuss.elastic.co/t/harvester-in-loop-while-parsing-json-data/220612/7 "2020-02-25T04:39:44Z")
I deleted the data and logs folder in Filebeat, the logs are being created.
This time I tried to send non-JSON logs but still it didn't send those.
This issue I am facing is on Windows 10 64-bit with filebeat 7.6.0, whereas with the same configuration and versions on linux OS, I am not facing this kind of issue, can you suggest me some stable version of filebeat for Windows 10?
---
### Author:  [@aksroh](https://discuss.elastic.co/u/aksroh)
#### Post date: [February 26, 2020, 4:34am UTC](https://discuss.elastic.co/t/harvester-in-loop-while-parsing-json-data/220612/8 "2020-02-26T04:34:30Z")
Found the solution, after the one line json, newline (\n) was missing in the file, when added, it worked properly on windows too.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [March 25, 2020, 4:34am UTC](https://discuss.elastic.co/t/harvester-in-loop-while-parsing-json-data/220612/9 "2020-03-25T04:34:51Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.