Hi,
We have issue while starting filebeat. It was not sending data to logstash after working successfully for around 30 days.
We checked that the size of registry file is 261MB. We are parsing around 1000 log files in this. May be this is the issue ? Do we need to configure something to flush ? Same configuration of filebeat is working in other servers.
Below are the errors. But we see process is running with ps -ef | grep filebeat
2018-03-01T12:32:17+08:00 INFO Publisher name: sdpprdamqls008
2018-03-01T12:32:17+08:00 INFO Flush Interval set to: 1s
2018-03-01T12:32:17+08:00 INFO Max Bulk Size set to: 2048
2018-03-01T12:32:17+08:00 INFO filebeat start running.
2018-03-01T12:32:17+08:00 INFO Registry file set to: /amdocs/apps/capplan/ELK/filebeat-5.1.1-linux-x86_64/data/registry
2018-03-01T12:32:17+08:00 INFO Loading registrar data from /amdocs/apps/capplan/ELK/filebeat-5.1.1-linux-x86_64/data/registry
2018-03-01T12:32:17+08:00 INFO Metrics logging every 30s
2018-03-01T12:32:18+08:00 INFO Stopping filebeat
2018-03-01T13:49:24+08:00 INFO No non-zero metrics in the last 30s
2018-03-01T13:49:38+08:00 INFO States Loaded from registrar: 1223290
2018-03-01T13:49:38+08:00 INFO Loading Prospectors: 2
2018-03-01T13:49:38+08:00 INFO Starting Registrar
2018-03-01T13:49:38+08:00 INFO Start sending events to output
2018-03-01T13:49:38+08:00 INFO Starting spooler: spool_size: 2048; idle_timeout: 5s
2018-03-01T13:49:39+08:00 INFO Prospector outlet closed
2018-03-01T13:49:39+08:00 ERR Problem putting initial state: prospector outlet closed
2018-03-01T13:49:39+08:00 INFO Stopping spooler
2018-03-01T13:49:39+08:00 INFO Stopping Registrar
2018-03-01T13:49:39+08:00 INFO Ending Registrar
2018-03-01T13:49:45+08:00 INFO Total non-zero values: registar.states.current=1223290 registrar.writes=1
2018-03-01T13:49:45+08:00 INFO Uptime: 1h17m28.168998869s
2018-03-01T13:49:45+08:00 INFO filebeat stopped.
2018-03-01T13:49:45+08:00 CRIT Exiting: Error in initing prospector: prospector outlet closed
In filebeat output we see below.
2018/03/01 07:35:28.203359 logp.go:230: INFO Non-zero metrics in the last 30s: publish.events=2516 registrar.states.update=2539 registrar.writes=2
2018/03/01 07:35:58.203322 logp.go:230: INFO Non-zero metrics in the last 30s: registrar.writes=2 registrar.states.update=2516 publish.events=2575
2018/03/01 07:36:28.203366 logp.go:230: INFO Non-zero metrics in the last 30s: registrar.states.update=2138 registrar.writes=2 publish.events=2476
2018/03/01 07:36:58.203307 logp.go:230: INFO Non-zero metrics in the last 30s: registrar.writes=2 publish.events=2407 registrar.states.update=1742
2018/03/01 07:37:28.203329 logp.go:230: INFO Non-zero metrics in the last 30s: registrar.states.update=2364 publish.events=1178 registrar.writes=1
2018/03/01 07:37:58.203361 logp.go:230: INFO Non-zero metrics in the last 30s: registrar.writes=2 registrar.states.update=2256 publish.events=2459
2018/03/01 07:38:28.203402 logp.go:230: INFO Non-zero metrics in the last 30s: publish.events=2395 registrar.states.update=2595 registrar.writes=2
2018/03/01 07:38:58.203337 logp.go:230: INFO Non-zero metrics in the last 30s: registrar.writes=2 registrar.states.update=2395 publish.events=2430
2018/03/01 07:39:28.203328 logp.go:230: INFO Non-zero metrics in the last 30s: registrar.writes=2 publish.events=2410 registrar.states.update=2085
2018/03/01 07:39:58.203346 logp.go:230: INFO Non-zero metrics in the last 30s: publish.events=2504 registrar.states.update=2410 registrar.writes=2
2018/03/01 07:40:28.203393 logp.go:230: INFO Non-zero metrics in the last 30s: publish.events=2429 registrar.states.update=2671 registrar.writes=2
2018/03/01 07:40:58.203362 logp.go:230: INFO Non-zero metrics in the last 30s: publish.events=2546 registrar.states.update=2572 registrar.writes=2
2018/03/01 07:41:28.203303 logp.go:230: INFO Non-zero metrics in the last 30s: registrar.writes=2 registrar.states.update=2581 publish.events=2548
2018/03/01 07:41:58.203327 logp.go:230: INFO Non-zero metrics in the last 30s: publish.events=2377 registrar.writes=2 registrar.states.update=2548
2018/03/01 07:42:28.203284 logp.go:230: INFO Non-zero metrics in the last 30s: registrar.writes=2 publish.events=2409 registrar.states.update=2377
2018/03/01 07:42:58.203314 logp.go:230: INFO Non-zero metrics in the last 30s: publish.events=2155 registrar.states.update=2409 registrar.writes=2