# Error when reloading prospectors for filebeat **URL:** \ **Category:** Beats\ **Created:** [April 20, 2017, 3:38pm UTC](https://discuss.elastic.co/t/error-when-reloading-prospectors-for-filebeat/83082 "2017-04-20T15:38:59Z")\ **Posts on this page:** 10\ **Page:** 1
**Author:** ![grantwu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/grantwu/32/15807_2.png) [@grantwu](https://discuss.elastic.co/u/grantwu)\ **Post date:** [April 20, 2017, 3:38pm UTC](https://discuss.elastic.co/t/error-when-reloading-prospectors-for-filebeat/83082/1 "2017-04-20T15:38:59Z")
The re-creation of a prospector as a condition for checking whether or not to stop or start a processor in the live reloading path causes prospectors to shut down due to error thrown when trying to load file state. This line: > Calls this function: [https://github.com/elastic/beats/blob/master/filebeat/prospector/factory.go#L27](https://github.com/elastic/beats/blob/master/filebeat/prospector/factory.go#L27) The factory function tries to call LoadStates on the prospector and if there is already a prospector running for the files in question, then the factory will return an error. Specifically this error: [https://github.com/elastic/beats/blob/master/filebeat/prospector/prospector\_log.go#L55](https://github.com/elastic/beats/blob/master/filebeat/prospector/prospector_log.go#L55) This causes the loop in the reload to continue which then causes the currently running prospector to be incorrectly shutdown once the loop is exited. For reference: I am currently running Filebeat on version 5.3 on CentOS6.9 ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [April 21, 2017, 11:02am UTC](https://discuss.elastic.co/t/error-when-reloading-prospectors-for-filebeat/83082/2 "2017-04-21T11:02:02Z")
As far as I understand you have the following setup: 1. Prospector 1 running 2. Prospector 2 config is moved into reload dir, prospector 1 is removed 3. Reloading happens 4. Prospector 2 does not start because of Finished, Prospector 1 stops That is currently the intended behaviour, as the next time the reload happens, it will start Prospector 2. Filebeat must guarantee that at one time no two prospectors with the same harvesters are running, otherwise you will have duplicates events and probably lots of strange side affects. ---
**Author:** ![grantwu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/grantwu/32/15807_2.png) [@grantwu](https://discuss.elastic.co/u/grantwu)\ **Post date:** [April 21, 2017, 4:20pm UTC](https://discuss.elastic.co/t/error-when-reloading-prospectors-for-filebeat/83082/3 "2017-04-21T16:20:52Z")
It's actually something more like, Prospector 1 and 2 are running and harvesting different files, either is removed, then both are stopped indefinitely. I don't believe I understand what you mean by the next time the reload happens since reload only happens when there's an update in the prospector config folder. This means the remaining prospector that should still be running could be stopped until another update is made which doesn't seem correct. I believe adding a 3rd prospector with a different harvester would actually cause 1 and 2 to stop as well. ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [April 24, 2017, 8:08am UTC](https://discuss.elastic.co/t/error-when-reloading-prospectors-for-filebeat/83082/4 "2017-04-24T08:08:35Z")
That actually sounds like a bug. If Prospector 2 cannot be started, it should not be added to the registry that keeps track on which prospectors are already running. Means it should be started during the next scan. There are potentially two issues here: - If a prospector is not started properly, it is still added to the registry but shouldn't - If a prospector is not started properly, it will not be started again because the scan for new files will return that no files were updated. Could you open a Github issue for this? ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [April 24, 2017, 8:10am UTC](https://discuss.elastic.co/t/error-when-reloading-prospectors-for-filebeat/83082/5 "2017-04-24T08:10:50Z")
Looking at the code again, I think point 1 should not apply. If there is an error during loading the state, it will not be added to the registry. Point 2 still applies. But that means if you add a Prospector 3 and Prospector 1 was stopped in the meantime, Prospector 2 and 3 should be started. ---
**Author:** ![grantwu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/grantwu/32/15807_2.png) [@grantwu](https://discuss.elastic.co/u/grantwu)\ **Post date:** [April 24, 2017, 5:54pm UTC](https://discuss.elastic.co/t/error-when-reloading-prospectors-for-filebeat/83082/6 "2017-04-24T17:54:32Z")
The issue isn't around the first time a prospector gets started. That logic is fine. The issue is around how the code determines which currently active prospectors/runners should continue to run after a configuration update. The code tries to execute logic that implies if a prospector is already running and has been unchanged in an update, the new prospector's hash/ID will match an element in the registry and thus the active prospector will be removed from the stop list. This unfortunately does not fully work. The main flaw is around how the factory attempts to load file states before checking if an active prospector matches a prospector defined in the updated configuration. The attempt to load file states will cause an error which means the reload loop will execute a continue and the prospector that should remain running after the reload is stopped since it is never removed from the stop list. ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [April 27, 2017, 12:24pm UTC](https://discuss.elastic.co/t/error-when-reloading-prospectors-for-filebeat/83082/7 "2017-04-27T12:24:06Z")
Ok, I see your point. Sorry that I missed that before. I need to check why our test do not cover this. We probably have to change where the loading of the state happen. One potential quick fix could be to check inside the error in reload if the prospector id already exists, and if yes, remove it from the delete list. Could you open a Github issue with that? ---
**Author:** ![ruflin](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ruflin/32/3116_2.png) [@ruflin](https://discuss.elastic.co/u/ruflin)\ **Post date:** [April 27, 2017, 1:20pm UTC](https://discuss.elastic.co/t/error-when-reloading-prospectors-for-filebeat/83082/8 "2017-04-27T13:20:32Z")
I opened [https://github.com/elastic/beats/pull/4128](https://github.com/elastic/beats/pull/4128) with a potential fix. I still need to add tests and check it in some more details. ---
**Author:** ![grantwu](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/grantwu/32/15807_2.png) [@grantwu](https://discuss.elastic.co/u/grantwu)\ **Post date:** [April 27, 2017, 9:50pm UTC](https://discuss.elastic.co/t/error-when-reloading-prospectors-for-filebeat/83082/9 "2017-04-27T21:50:11Z")
Thanks! Here's the github issue: [https://github.com/elastic/beats/issues/4133](https://github.com/elastic/beats/issues/4133) ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 11, 2017, 3:49pm UTC](https://discuss.elastic.co/t/error-when-reloading-prospectors-for-filebeat/83082/10 "2017-05-11T15:49:58Z")
This topic was automatically closed after 21 days. New replies are no longer allowed.