i've added to log files to be shipped, if i do them individually no problem if i try and send both - only one of them end up in elasticsearch ? what gives?
015/10/15 15:11:41.183630 Loading registrar data from /opt/logstash-forwarder/bin/.logstash-forwarder
2015/10/15 15:11:41.183707 Waiting for 1 prospectors to initialise
2015/10/15 15:11:41.183787 Resuming harvester on a previously harvested file: /var/log/cisco.log
2015/10/15 15:11:41.183817 Registrar will re-save state for /var/log/cisco.log
2015/10/15 15:11:41.183827 Registrar will re-save state for /var/log/cisco2.log
2015/10/15 15:11:41.183866 harvest: "/var/log/cisco.log" position:138027076 (offset snapshot:138027076)
2015/10/15 15:11:41.183923 Resuming harvester on a previously harvested file: /var/log/cisco2.log
2015/10/15 15:11:41.183968 harvest: "/var/log/cisco2.log" position:356308108 (offset snapshot:356308108)
2015/10/15 15:11:41.184193 All prospectors initialised with 2 states to persist
2015/10/15 15:11:41.184582 Setting trusted CA from file: /etc/pki/tls/certs/logstash-forwarder.crt
2015/10/15 15:11:41.184974 Connecting to [172.17.20.201]:5000 (172.17.20.201)
Are you getting entries from cisco.log but not cisco2.log? Presumably you're getting all new entries from the former file but nothing from the latter since it isn't changing. If you want to reprocess files you need to adjust or delete LSF's state file (.logstash-forwarder).
did a delete of the file, I've made a viz of the number of entries in the individial logs and give it some time to run - it should show if all the logs are running ok - one may me more busy than the other. thank you for your quick reply!
the problem is still there im affraid. it ran ok over the weekend but then stops oddly enough at he same time a log rotation takes place - but only for the 2nd logfile mentioned in the conf. file. the first log file cisco.log runs fine with out a problem - through the log rotation.
If i restart the logstash forwarder it will send the second log (cisco2.log) for a while and then stop but the first file cisco.log keeps going without a problem. there are no entries in the log files for the logstash forwarder or the .err file.
the cisco.log file is alot more active than the cisco2 log file - is this a problem it seams like it "downs out" the cisco2.log files data. have you seen similar behavior anywhere else? would it help if i split up the logfiles and sendt them on differrnt ports to the logstash server?
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.