Logstash-Forwarder stopping registrar of logs from access log after reandom time


(Vineeth Zacharia) #1

Hi,

I am trying to do a simple ELK stack configuration with pulling of logs from log-stash forwarder from one of the apache servers. I am running 0.4.0 version of logstash-forwarder. It randomly stops to pull logs from the apache logs after some time. Following is my configuration -
--- options -------
config-arg: /etc/logstash-forwarder.conf
idle-timeout: 5s
spool-size: 1024
harvester-buff-size: 16384
--- flags ---------
tail (on-rotation): true
log-to-syslog: false
quiet: false
{
"network": {
"servers": [ "<Server_HOSTNAME>:5043" ],
"ssl ca": "/etc/ssl/logstash-forwarder.crt",
"ssl key": "/etc/ssl/logstash-forwarder.key"
},

The list of files configurations

"files": [
{
"paths": [ "/etc/httpd/logs/access_log*" ],
"fields": { "type": "apache", "tail": "true" }
}
]
}

2015/08/27 15:41:28.968770 Loading registrar data from /opt/logstash-forwarder/bin/.logstash-forwarder
2015/08/27 15:41:28.968898 Waiting for 1 prospectors to initialise
2015/08/27 15:41:28.969432 Resuming harvester on a previously harvested file: /etc/httpd/logs/access_log
2015/08/27 15:41:28.969471 Skipping file (older than dead time of 24h0m0s): /etc/httpd/logs/access_log.1.gz
2015/08/27 15:41:28.969498 Skipping file (older than dead time of 24h0m0s): /etc/httpd/logs/access_log.2.gz
2015/08/27 15:41:28.969524 Skipping file (older than dead time of 24h0m0s): /etc/httpd/logs/access_log.3.gz
2015/08/27 15:41:28.969549 Skipping file (older than dead time of 24h0m0s): /etc/httpd/logs/access_log.4.gz
2015/08/27 15:41:28.969566 Registrar will re-save state for /etc/httpd/logs/access_log
2015/08/27 15:41:28.969579 All prospectors initialised with 1 states to persist
2015/08/27 15:41:28.969704 harvest: "/etc/httpd/logs/access_log" position:27034616 (offset snapshot:27034616)
2015/08/27 15:41:28.970528 Setting trusted CA from file: /etc/ssl/logstash-forwarder.crt
2015/08/27 15:41:28.973650 Connecting to [IP]:5043 (ELK Server)
2015/08/27 15:41:29.256386 Connected to IP
2015/08/27 15:41:33.252127 Registrar: processing 1024 events
2015/08/27 15:41:35.447260 Registrar: processing 1024 events
2015/08/27 15:41:37.436426 Registrar: processing 654 events
2015/08/27 15:41:51.478578 Registrar: processing 1 events
2015/08/27 15:42:18.981019 Registrar: processing 1 events

What configuration I am missing ?? I am starting logstash-forwarder like this -

nohup ./logstash-forwarder --config="/etc/logstash-forwarder.conf" -tail=true &


(system) #2