Filebeat file-god stopped and No modules or prospectors enabled and configuration reloading disabled

[caoimhe]

Service featbeat start exits with message "Exiting: No modules or prospectors enabled and configuration reloading disabled. What files do you want me to watch?"
Service filebeat status returns filebeat-god has stopped.
Service filebeat stop fails

How do I enable modules or prospectors and enable configuration reloading?
Is this linked to the file-god issue?

Included below is the error messages I'm receiving and my filebeat.yml

Thanks,
Caoimhe

service filebeat start

Starting filebeat: 2018/04/19 15:41:14.111768 beat.go:297: INFO Home path: [/usr/share/filebeat] Config path: [/etc/filebeat] Data path: [/data/filebeat] Logs path: [/var/log/filebeat]
2018/04/19 15:41:14.111814 beat.go:192: INFO Setup Beat: filebeat; Version: 5.6.2
2018/04/19 15:41:14.111855 publish.go:228: WARN Support for loading more than one output is deprecated and will not be supported in version 6.0.
2018/04/19 15:41:14.111903 metrics.go:23: INFO Metrics logging every 30s
2018/04/19 15:41:14.112080 output.go:258: INFO Loading template enabled. Reading template file: /etc/filebeat/filebeat.template.json
2018/04/19 15:41:14.113092 output.go:269: INFO Loading template enabled for Elasticsearch 2.x. Reading template file: /etc/filebeat/filebeat.template-es2x.json
2018/04/19 15:41:14.114016 output.go:281: INFO Loading template enabled for Elasticsearch 6.x. Reading template file: /etc/filebeat/filebeat.template-es6x.json
2018/04/19 15:41:14.114926 client.go:128: INFO Elasticsearch url: http://localhost:9200
2018/04/19 15:41:14.114975 outputs.go:108: INFO Activated elasticsearch as output plugin.
2018/04/19 15:41:14.115027 logstash.go:90: INFO Max Retries set to: 3
2018/04/19 15:41:14.115112 outputs.go:108: INFO Activated logstash as output plugin.
2018/04/19 15:41:14.115228 publish.go:300: INFO Publisher name: sd-05cf-0367
2018/04/19 15:41:14.115406 async.go:63: INFO Flush Interval set to: 1s
2018/04/19 15:41:14.115421 async.go:64: INFO Max Bulk Size set to: 50
2018/04/19 15:41:14.115480 async.go:63: INFO Flush Interval set to: 1s
2018/04/19 15:41:14.115489 async.go:64: INFO Max Bulk Size set to: 2048
2018/04/19 15:41:14.115717 beat.go:346: CRIT Exiting: No modules or prospectors enabled and configuration reloading disabled. What files do you want me to watch?
Exiting: No modules or prospectors enabled and configuration reloading disabled. What files do you want me to watch?

service filebeat status
filebeat-god is stopped

service filebeat stop
Stopping filebeat: [FAILED]

cat filebeat.yml

filebeat.prospectors:
 paths:
  - /etc/TFSlogs/*.log
 input_type: log
 enabled: true
 document_type: log

  #-C:\ProgramData\Microsoft\Team Foundation\Server Configuration\Logs\*
  #- /var/log/messages
  #- c:\programdata\elasticsearch\logs\*
 
fields:
 environment: dev
 hosting_env: citi
 log_type: syslog
#================================ Outputs =====================================

Configure what outputs to use when sending the data collected by the beat
#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
 hosts: ["localhost:9200"]


#----------------------------- Logstash output --------------------------------
output.logstash:
 hosts: ["localhost:5044"]

loadbalance: true

#================================ Logging =====================================

logging.to_files: true
logging.to_syslog: false

[harshbajaj16 (Harsh Bajaj)]

Hi caoimhe,

@caoimhe

filebeat.prospectors:
paths:
/etc/TFSlogs/*.log
input_type: log
enabled: true
document_type: log

It seems your config file have some error as you defined enabled: true after paths: which is not correct.

Solution :
could you please try with below format :

filebeat.prospectors:

• type: log
  enabled: true
  paths:
• /etc/TFSlogs/*.log

Thanks,
Harsh Bajaj

[caoimhe]

Hi Harsh,

Thanks for the suggestion. This fixed it and filebeat is now running, however my Kibana is not showing the logs from the path: /etc/TFSlogs/*.log. Instead it is displaying old log files from the path /var/log/messages.

Do you have any idea why this might be?

Thanks,
Caoimhe

[harshbajaj16 (Harsh Bajaj)]

Hi,

There might be so many reasons :

You can check with below suggestions :

Please check either you have applied any filter in kibana or check whether latest logs are coming from filebeat to logstash and also check elasticsearch index.

Restart services and check but first you should go with above suggestion. If your services are working fine then your latest logs should be visible in kibana (check kibana dashboard settings).

Thanks,
Harsh Bajaj

[caoimhe]

Hi,

Thanks for your help! I got it working live using -var/log/messages. However, I have static log files in - /etc/TFSlogs are not being read.

I'm not sure that I'm using the index in the correctly in logstash.

[Dev root @ sd-05cf-0378 /etc/logstash/conf.d]
cat logstash-simple.conf

input{
beats{
port=>5044
}
}

output {
elasticsearch{
index=>"logstash-simple.conf"
template_overwrite=>"true"
template_name=>"logstash-simple.conf"
hosts=>["http://localhost:9200"]
}
}

Cheers,
Caoimhe

[harshbajaj16 (Harsh Bajaj)]

Hi Caoimhe,

Could you please share the logs of filebeat and also share the config file of filebeat so that we can identify the issue.

Log file path : /var/log/filebeat/
Config file : /etc/filebeat/filebeat.yml

Thanks,
Harsh Bajaj

[caoimhe]

Hi,

I got it connected, I forgot to add the /* to the path -/etc/TFSlogs/*.
Thanks for all your help

This is my filebeat.yml file which worked:

Filebeat.yml

# cat filebeat.yml
#=========================== Filebeat prospectors =============================

filebeat.prospectors:
- type: log
  enabled: true
  paths:
  - /etc/TFSlogs/*
 #- /var/log/messages
 
fields:
  environment: dev
  hosting_env: citi
  log_type: syslog
#================================ Outputs =====================================

# Configure what outputs to use when sending the data collected by the beat
#-------------------------- Elasticsearch output ------------------------------
output.elasticsearch:
  hosts: ["localhost:9200"]


#----------------------------- Logstash output --------------------------------
output.logstash:
  hosts: ["localhost:5044"]

#loadbalance: true

#================================ Logging =====================================

logging.to_files: true
#logging.to_syslog: false

Would the formatting of the log file account for why it would not be pick up?