I am getting the error same as the Title. Filebeat is not running. What is the reason and how to cure this problem ?
It seems you have the same file defined in 2 different prospectors. Can you share your filebeat config and the version?
I'm having the same problem... My filebeat.yml file is like this:
filebeat:
config_dir: "C:/ELK/filebeat"
registry_file: "C:/ELK/filebeat/registry"
prospectors:
- input_type: log
paths:
- D:\inetpub\logs\logfiles\W3SVC2\*.log
document_type: iis_log
output:
logstash:
hosts:
- my.logstash.server.net:5044
logging:
level: "error"
files:
rotateeverybytes: 10485760
I was having this problem with filebeat 5.3.2, so I upgraded to 5.4.0. It ran once and filled some values into the registry file, but crashed, and now always gives this error:
C:\ELK\filebeat>filebeat.exe -e -c C:\elk\filebeat\filebeat.yml
2017/05/11 20:25:21.178825 beat.go:339: CRIT Exiting: Prospector with same ID al
ready exists: 3814112069515792729
Exiting: Prospector with same ID already exists: 3814112069515792729
The ID changes if I make a meaningful change to the configuration...
I've tried these paths (only one at a time):
- D:\inetpub\logs\logfiles**.log
- D:\inetpub\logs\logfiles\W3SVC2*.log
- D:\inetpub\logs\logfiles\W3SVC2\u_ex170511.log
..each time, the same error that "ID already exists".
(There are no "filebeat" processes running)
I also have deleted the registry file and re-run with those various paths, and still, the same error (though after deleting the registry file, it never once has filled the registry file with any meaningful information... it always now gets created again with the content: []
As far as I can tell, I've only defined one prospector... What could be going wrong?
I discovered the "DEBUG" log level, and it shows that it does think there are 2 prospectors configured. I deleted all blank lines from the filebeat.yml file and ran again, with no improvement.
The line "Additional configs loaded from" seems to indicate that it may be loading the config twice, so I ran it with out the "-c C:\elk\filebeat\filebeat.yml" flag, but it still gets the same error and still says "Additional configs loaded from".
Here is the debug log:
C:\ELK\filebeat>filebeat.exe -e -c C:\elk\filebeat\filebeat.yml
2017/05/11 21:31:49.528448 beat.go:285: INFO Home path: [C:\ELK\filebeat] Config path: [C:\ELK\filebeat] Data path: [C:\
ELK\filebeat\data] Logs path: [C:\ELK\filebeat\logs]
2017/05/11 21:31:49.530401 beat.go:186: INFO Setup Beat: filebeat; Version: 5.4.0
2017/05/11 21:31:49.530401 processor.go:44: DBG Processors:
2017/05/11 21:31:49.531378 beat.go:192: DBG Initializing output plugins
2017/05/11 21:31:49.531378 logstash.go:90: INFO Max Retries set to: 3
2017/05/11 21:31:49.532354 outputs.go:108: INFO Activated logstash as output plugin.
2017/05/11 21:31:49.533331 publish.go:238: DBG Create output worker
2017/05/11 21:31:49.533331 publish.go:280: DBG No output is defined to store the topology. The server fields might not
be filled.
2017/05/11 21:31:49.534307 publish.go:295: INFO Publisher name: XXXXXXXXX
2017/05/11 21:31:49.538213 metrics.go:23: INFO Metrics logging every 30s
2017/05/11 21:31:49.542119 async.go:63: INFO Flush Interval set to: 1s
2017/05/11 21:31:49.542119 async.go:64: INFO Max Bulk Size set to: 2048
2017/05/11 21:31:49.543096 async.go:72: DBG create bulk processing worker (interval=1s, bulk size=2048)
2017/05/11 21:31:49.544072 config.go:114: INFO Additional config files are fetched from: C:/ELK/filebeat
2017/05/11 21:31:49.544072 config.go:87: INFO Additional configs loaded from: C:\ELK\filebeat\filebeat.yml
2017/05/11 21:31:49.546025 beat.go:221: INFO filebeat start running.
2017/05/11 21:31:49.546025 registrar.go:85: INFO Registry file set to: C:/ELK/filebeat/registry
2017/05/11 21:31:49.546025 service_windows.go:51: DBG Windows is interactive: true
2017/05/11 21:31:49.547002 registrar.go:106: INFO Loading registrar data from C:/ELK/filebeat/registry
2017/05/11 21:31:49.548955 registrar.go:123: INFO States Loaded from registrar: 0
2017/05/11 21:31:49.548955 crawler.go:38: INFO Loading Prospectors: 2
2017/05/11 21:31:49.549931 prospector.go:83: DBG File Configs: [D:\inetpub\logs\logfiles\*\*.log]
2017/05/11 21:31:49.550908 prospector_log.go:44: DBG exclude_files: []
2017/05/11 21:31:49.550908 prospector_log.go:65: INFO Prospector with previous states loaded: 0
2017/05/11 21:31:49.551884 prospector.go:124: INFO Starting prospector of type: log; id: 7832652165051331537
2017/05/11 21:31:49.552861 prospector.go:83: DBG File Configs: [D:\inetpub\logs\logfiles\*\*.log]
2017/05/11 21:31:49.553837 crawler.go:90: INFO Stopping Crawler
2017/05/11 21:31:49.553837 crawler.go:100: INFO Stopping 1 prospectors
2017/05/11 21:31:49.549931 sync.go:41: INFO Start sending events to output
2017/05/11 21:31:49.549931 spooler.go:63: INFO Starting spooler: spool_size: 2048; idle_timeout: 5s
2017/05/11 21:31:49.552861 prospector_log.go:70: DBG Start next scan
2017/05/11 21:31:49.549931 registrar.go:236: INFO Starting Registrar
2017/05/11 21:31:49.559696 prospector_log.go:216: INFO Scan aborted because prospector stopped.
2017/05/11 21:31:49.559696 prospector_log.go:91: DBG Prospector states cleaned up. Before: 0, After: 0
2017/05/11 21:31:49.560673 prospector.go:180: INFO Prospector ticker stopped
2017/05/11 21:31:49.561649 prospector.go:232: INFO Stopping Prospector: 7832652165051331537
2017/05/11 21:31:49.562626 prospector.go:134: INFO Prospector channel stopped
2017/05/11 21:31:49.562626 crawler.go:112: INFO Crawler stopped
2017/05/11 21:31:49.562626 spooler.go:101: INFO Stopping spooler
2017/05/11 21:31:49.563602 spooler.go:109: DBG Spooler has stopped
2017/05/11 21:31:49.564579 sync.go:47: DBG Shutting down sync publisher
2017/05/11 21:31:49.565555 registrar.go:291: INFO Stopping Registrar
2017/05/11 21:31:49.565555 registrar.go:248: INFO Ending Registrar
2017/05/11 21:31:49.566532 registrar.go:298: DBG Write registry file: C:/ELK/filebeat/registry
2017/05/11 21:31:49.568485 registrar.go:323: DBG Registry file updated. 0 states written.
2017/05/11 21:31:49.570438 metrics.go:51: INFO Total non-zero values: registrar.writes=1
2017/05/11 21:31:49.570438 metrics.go:52: INFO Uptime: 46.872ms
2017/05/11 21:31:49.571414 beat.go:225: INFO filebeat stopped.
2017/05/11 21:31:49.571414 beat.go:339: CRIT Exiting: Prospector with same ID already exists: 7832652165051331537
Exiting: Prospector with same ID already exists: 7832652165051331537The "Additional Config Files were loaded" was the clue I needed...
I moved the filebeat.yml file to a different folder (C:\ELK), then ran with the flag "-c C:\elk\filebeat.yml" and it ran!
The fact that running "filebeat.exe -e" (when the filebeat.yml file was at C:\elk\filebeat\filebeat.yml) resulted in the error seems to me to indicate there is some bug.
Can you share both the config files? I assume one is overloading the other. The config files in config_dir should only contain prospector configurations.
Don't want to hijack this thread, but same issue occurred for me, Filebeat 5.4.0 refuses to start.
I do have lots of prospectors, but that was not an issue in previous versions of Filebeat (4.x-5.3.1).
my filebeat.yml is way to large too paste into the 7000 character limit of this post.
[SOLVED]: It turns out I did have duplicate lines, which wasn't a fatal issue prior to 5.4.0. I used the following command to find what lines were duplicates:
sort filebeat.yml | uniq -cd | sort -nr
Check to see if you have "overlapping" glob patterns such that the same file is picked up by more than one prospector.
Also make sure that if you are using the conf.d feature that you haven't inadvertently set the dir to the same location as where your main filebeat.yml is located.
1 Like
Thank you for your thoughts...
I already shared my one filebeat.yml configuration file in my first post... there are not two files.
The difference is that if that file exists in C:\ELK\filebeat then the error happens.
If the file is moved into C:\ELK then filebeat is able to launch with no problem.
That is because of this:
You have enabled the config_dir feature AND you are storing your filebeat.yml in that same directory. So Filebeat is loading your config twice. If you aren't making use the config_dir feature then just remove that from your config.
Ah! That explains it completely, thank you!
Is that documented anywhere? I've read all the "configuring filebeat" pages I could find and they didn't make it clear where filebeat will look to find config files. In my experience with other server daemons, if you specify a config file on the command line, that overrides any auto-found/auto-loaded configs... it never occurred to me that it would instead "add" it.
I even did read the "filebeat --help" information, which says:
-c value
Configuration file, relative to path.config (default filebeat.yml)
Saying "default filebeat.yml" indicates to me that the -c value will override that default, not add to it.
So, thank you very much for explaining how it actually works!
Using -c does override the default config file location. It's not really the -c option that's causing the issue. It's the config_dir setting in your config file causing the problem. This setting is used to have a conf.d directory where Filebeat reads config snippets (like one config file for nginx logs, one for mysql logs, etc.).
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.