CRIT Exiting: Error in initing prospector: prospector outlet closed

[bismayam (Bismaya)]

Hi,

We have issue while starting filebeat. It was not sending data to logstash after working successfully for around 30 days.

We checked that the size of registry file is 261MB. We are parsing around 1000 log files in this. May be this is the issue ? Do we need to configure something to flush ? Same configuration of filebeat is working in other servers.

Below are the errors. But we see process is running with ps -ef | grep filebeat

2018-03-01T12:32:17+08:00 INFO Publisher name: sdpprdamqls008
2018-03-01T12:32:17+08:00 INFO Flush Interval set to: 1s
2018-03-01T12:32:17+08:00 INFO Max Bulk Size set to: 2048
2018-03-01T12:32:17+08:00 INFO filebeat start running.
2018-03-01T12:32:17+08:00 INFO Registry file set to: /amdocs/apps/capplan/ELK/filebeat-5.1.1-linux-x86_64/data/registry
2018-03-01T12:32:17+08:00 INFO Loading registrar data from /amdocs/apps/capplan/ELK/filebeat-5.1.1-linux-x86_64/data/registry
2018-03-01T12:32:17+08:00 INFO Metrics logging every 30s
2018-03-01T12:32:18+08:00 INFO Stopping filebeat
2018-03-01T13:49:24+08:00 INFO No non-zero metrics in the last 30s
2018-03-01T13:49:38+08:00 INFO States Loaded from registrar: 1223290
2018-03-01T13:49:38+08:00 INFO Loading Prospectors: 2
2018-03-01T13:49:38+08:00 INFO Starting Registrar
2018-03-01T13:49:38+08:00 INFO Start sending events to output
2018-03-01T13:49:38+08:00 INFO Starting spooler: spool_size: 2048; idle_timeout: 5s
2018-03-01T13:49:39+08:00 INFO Prospector outlet closed
2018-03-01T13:49:39+08:00 ERR Problem putting initial state: prospector outlet closed
2018-03-01T13:49:39+08:00 INFO Stopping spooler
2018-03-01T13:49:39+08:00 INFO Stopping Registrar
2018-03-01T13:49:39+08:00 INFO Ending Registrar
2018-03-01T13:49:45+08:00 INFO Total non-zero values: registar.states.current=1223290 registrar.writes=1
2018-03-01T13:49:45+08:00 INFO Uptime: 1h17m28.168998869s
2018-03-01T13:49:45+08:00 INFO filebeat stopped.
2018-03-01T13:49:45+08:00 CRIT Exiting: Error in initing prospector: prospector outlet closed

In filebeat output we see below.

2018/03/01 07:35:28.203359 logp.go:230: INFO Non-zero metrics in the last 30s: publish.events=2516 registrar.states.update=2539 registrar.writes=2
2018/03/01 07:35:58.203322 logp.go:230: INFO Non-zero metrics in the last 30s: registrar.writes=2 registrar.states.update=2516 publish.events=2575
2018/03/01 07:36:28.203366 logp.go:230: INFO Non-zero metrics in the last 30s: registrar.states.update=2138 registrar.writes=2 publish.events=2476
2018/03/01 07:36:58.203307 logp.go:230: INFO Non-zero metrics in the last 30s: registrar.writes=2 publish.events=2407 registrar.states.update=1742
2018/03/01 07:37:28.203329 logp.go:230: INFO Non-zero metrics in the last 30s: registrar.states.update=2364 publish.events=1178 registrar.writes=1
2018/03/01 07:37:58.203361 logp.go:230: INFO Non-zero metrics in the last 30s: registrar.writes=2 registrar.states.update=2256 publish.events=2459
2018/03/01 07:38:28.203402 logp.go:230: INFO Non-zero metrics in the last 30s: publish.events=2395 registrar.states.update=2595 registrar.writes=2
2018/03/01 07:38:58.203337 logp.go:230: INFO Non-zero metrics in the last 30s: registrar.writes=2 registrar.states.update=2395 publish.events=2430
2018/03/01 07:39:28.203328 logp.go:230: INFO Non-zero metrics in the last 30s: registrar.writes=2 publish.events=2410 registrar.states.update=2085
2018/03/01 07:39:58.203346 logp.go:230: INFO Non-zero metrics in the last 30s: publish.events=2504 registrar.states.update=2410 registrar.writes=2
2018/03/01 07:40:28.203393 logp.go:230: INFO Non-zero metrics in the last 30s: publish.events=2429 registrar.states.update=2671 registrar.writes=2
2018/03/01 07:40:58.203362 logp.go:230: INFO Non-zero metrics in the last 30s: publish.events=2546 registrar.states.update=2572 registrar.writes=2
2018/03/01 07:41:28.203303 logp.go:230: INFO Non-zero metrics in the last 30s: registrar.writes=2 registrar.states.update=2581 publish.events=2548
2018/03/01 07:41:58.203327 logp.go:230: INFO Non-zero metrics in the last 30s: publish.events=2377 registrar.writes=2 registrar.states.update=2548
2018/03/01 07:42:28.203284 logp.go:230: INFO Non-zero metrics in the last 30s: registrar.writes=2 publish.events=2409 registrar.states.update=2377
2018/03/01 07:42:58.203314 logp.go:230: INFO Non-zero metrics in the last 30s: publish.events=2155 registrar.states.update=2409 registrar.writes=2

[tag_v]

check "ignore_older " prospectors mentioned in conf. would be easy to figure out with conf shared

[ruflin]

It definitively seems like your registry has grown too large. Which filebeat version are you using? You should have a look at the clean_* options.

[bismayam (Bismaya)]

We are using filebeat 5.1.1. Is there any potential loss of data in case we configure clean_* : 1 ? We see different clean options ignores in some condition.

[bismayam (Bismaya)]

We splitted filebeat to different servers and reduced the logs . Now it is working fine.

[ruflin]

clean_* options should not lead to any data loss.

Glad do hear you got it working.