my configuration like the below
filebeat:
prospectors:
-
paths:
- /home/mohamed/localhost_access_log.2015-12-20.txt
input_type: syslog
registry_file: /var/lib/filebeat/registry
output:
elasticsearch:
enabled: false
hosts: ["localhost:9200"]
logstash:
hosts: ["logs.e-masary.net:5044 "]
tls:
certificate_authorities: ["/etc/pki/tls/certs/logstash-forwarder.crt"]
shipper:
logging:
files:
rotateeverybytes: 10485760 # = 10MB
i change the path to new location to start monitor it but flebeat still sent log from /var/log/*log
any help
simply i need to monitor tomcat logs ?
and i installed from this tutorial
In this tutorial, we will go over the installation of the Elasticsearch ELK Stack on CentOS 7—that is, Elasticsearch 2.1.x, Logstash 2.1.x, and Kibana 4.3.x. We will also show you how to configure it to gather and visualize the syslogs of your...
warkolm
(Mark Walkom)
January 3, 2016, 9:59pm
3
Did you restart filebeat after changing the path?
yes and logstash & elasticsearch
ruflin
(ruflin)
January 4, 2016, 7:41am
5
Can you run filebeat with the -e -d "*" flag and provide the output? In addition, it would be good to know the version of filebeat you are using and your OS.
root@mohamed:~# /etc/filebeat/filebeat.yml -e -d '*'
/etc/filebeat/filebeat.yml: line 4: filebeat:: command not found
/etc/filebeat/filebeat.yml: line 6: prospectors:: command not found
/etc/filebeat/filebeat.yml: line 8: -: command not found
/etc/filebeat/filebeat.yml: line 14: paths:: command not found
/etc/filebeat/filebeat.yml: line 15: -: command not found
/etc/filebeat/filebeat.yml: line 33: input_type:: command not found
/etc/filebeat/filebeat.yml: line 59: scan_frequency:: command not found
/etc/filebeat/filebeat.yml: line 132: registry_file:: command not found
/etc/filebeat/filebeat.yml: line 149: output:: command not found
/etc/filebeat/filebeat.yml: line 152: elasticsearch:: command not found
/etc/filebeat/filebeat.yml: line 153: enabled:: command not found
/etc/filebeat/filebeat.yml: line 158: hosts:: command not found
/etc/filebeat/filebeat.yml: line 231: logstash:: command not found
/etc/filebeat/filebeat.yml: line 233: hosts:: command not found
/etc/filebeat/filebeat.yml: line 246: tls:: command not found
/etc/filebeat/filebeat.yml: line 248: certificate_authorities:: command not found
/etc/filebeat/filebeat.yml: line 294: shipper:: command not found
/etc/filebeat/filebeat.yml: line 332: logging:: command not found
/etc/filebeat/filebeat.yml: line 343: files:: command not found
/etc/filebeat/filebeat.yml: line 352: rotateeverybytes:: command not found
ruflin
(ruflin)
January 4, 2016, 7:55am
8
Below is the command. Above you tried to run your config file.
filebeat -c /etc/filebeat/filbeat.yml -e -d "*"
mohamed@mohamed:~$ filebeat -c /etc/filebeat/filebeat.yml -e -d "*"
2016/01/04 08:16:26.958219 beat.go:97: DBG Initializing output plugins
2016/01/04 08:16:26.958249 geolite.go:24: INFO GeoIP disabled: No paths were set under output.geoip.paths
2016/01/04 08:16:26.958297 client.go:244: DBG ES Ping(url=http://localhost:9200, timeout=1m30s)
2016/01/04 08:16:26.958593 client.go:249: DBG Ping request failed with: Head http://localhost:9200 : dial tcp 127.0.0.1:9200: getsockopt: connection refused
2016/01/04 08:16:26.958612 outputs.go:111: INFO Activated elasticsearch as output plugin.
2016/01/04 08:16:27.037861 outputs.go:111: INFO Activated logstash as output plugin.
2016/01/04 08:16:27.037901 publish.go:198: DBG create output worker: 0x0, 0xc8200c8eb0
2016/01/04 08:16:27.037969 publish.go:198: DBG create output worker: 0x0, 0x0
2016/01/04 08:16:27.037993 publish.go:235: DBG No output is defined to store the topology. The server fields might not be filled.
2016/01/04 08:16:27.038037 publish.go:249: INFO Publisher name: mohamed
2016/01/04 08:16:27.038183 async.go:95: DBG create bulk processing worker (interval=1s, bulk size=50)
2016/01/04 08:16:27.038225 async.go:95: DBG create bulk processing worker (interval=1s, bulk size=200)
2016/01/04 08:16:27.038269 beat.go:107: INFO Init Beat: filebeat; Version: 1.0.1
2016/01/04 08:16:27.038915 beat.go:133: INFO filebeat sucessfully setup. Start running.
2016/01/04 08:16:27.038944 registrar.go:66: INFO Registry file set to: /var/lib/filebeat/registry
2016/01/04 08:16:27.038964 registrar.go:76: INFO Loading registrar data from /var/lib/filebeat/registry
2016/01/04 08:16:27.038997 spooler.go:44: DBG Set idleTimeoutDuration to 5s
2016/01/04 08:16:27.039018 crawler.go:38: DBG File Configs: [/home/mohamed/localhost_access_log.2015-12-20.txt]
2016/01/04 08:16:27.039036 prospector.go:128: DBG Set ignore_older duration to 24h0m0s
2016/01/04 08:16:27.039052 prospector.go:128: DBG Set scan_frequency duration to 10s
2016/01/04 08:16:27.039065 prospector.go:128: DBG Set backoff duration to 1s
2016/01/04 08:16:27.039078 prospector.go:128: DBG Set max_backoff duration to 10s
2016/01/04 08:16:27.039092 prospector.go:128: DBG Set partial_line_waiting duration to 5s
2016/01/04 08:16:27.039104 crawler.go:58: DBG Waiting for 1 prospectors to initialise
2016/01/04 08:16:27.039125 prospector.go:141: DBG Harvest path: /home/mohamed/localhost_access_log.2015-12-20.txt
2016/01/04 08:16:27.039154 prospector.go:207: DBG scan path /home/mohamed/localhost_access_log.2015-12-20.txt
2016/01/04 08:16:27.039176 prospector.go:219: DBG Check file for harvesting: /home/mohamed/localhost_access_log.2015-12-20.txt
2016/01/04 08:16:27.039191 prospector.go:273: DBG Start harvesting unknown file: /home/mohamed/localhost_access_log.2015-12-20.txt
2016/01/04 08:16:27.039206 prospector.go:289: DBG Fetching old state of file to resume: /home/mohamed/localhost_access_log.2015-12-20.txt
2016/01/04 08:16:27.039224 prospector.go:306: DBG Skipping file (older than ignore older of 24h0m0s, 351h2m55.007260138s): /home/mohamed/localhost_access_log.2015-12-20.txt
2016/01/04 08:16:27.039239 prospector.go:207: DBG scan path /home/mohamed/localhost_access_log.2015-12-20.txt
2016/01/04 08:16:27.039254 prospector.go:219: DBG Check file for harvesting: /home/mohamed/localhost_access_log.2015-12-20.txt
2016/01/04 08:16:27.039268 prospector.go:341: DBG Update existing file for harvesting: /home/mohamed/localhost_access_log.2015-12-20.txt
2016/01/04 08:16:27.039280 prospector.go:383: DBG Not harvesting, file didn't change: /home/mohamed/localhost_access_log.2015-12-20.txt
2016/01/04 08:16:27.039295 crawler.go:65: DBG No pending prospectors. Finishing setup
2016/01/04 08:16:27.039308 crawler.go:78: INFO All prospectors initialised with 0 states to persist
2016/01/04 08:16:27.039321 registrar.go:83: INFO Starting Registrar
2016/01/04 08:16:27.039346 filebeat.go:122: INFO Start sending events to output
2016/01/04 08:16:27.039411 spooler.go:77: INFO Starting spooler: spool_size: 1024; idle_timeout: 5s
2016/01/04 08:16:29.539524 spooler.go:97: DBG Flushing spooler because of timemout. Events flushed: 0
^C2016/01/04 08:16:34.698914 service.go:27: DBG Received sigterm/sigint, stopping
2016/01/04 08:16:34.698938 registrar.go:129: INFO Stopping Registrar
2016/01/04 08:16:34.698963 registrar.go:93: INFO Ending Registrar
2016/01/04 08:16:34.698979 registrar.go:142: DBG Write registry file: /var/lib/filebeat/registry
2016/01/04 08:16:34.699058 registrar.go:147: ERR Failed to create tempfile (/var/lib/filebeat/registry.new) for writing: open /var/lib/filebeat/registry.new: permission denied
2016/01/04 08:16:34.699073 beat.go:143: INFO Cleaning up filebeat before shutting down.
many thanks the problem was on the file, it's static file no changes has been don on this file when i copy a new file overwrite the old one the data come.
ruflin
(ruflin)
January 4, 2016, 9:17am
13
Good to hear your were able to solve the problem.