Logstash 2.1 is filling my Disk volumes

Elastic Stack Logstash
#1

Hi Team , I am facing an issue that when my logstash process is running for 4-5 days , my disk volume on the machine where logstash is installed is getting exhausted and i have to restart the logstash process to free the volume . Can anyone please let me know how logstash exhausts the volumes on the machine.

Volume when logstash was running :slight_smile:

root 61G 61G 32K 100% /

Volume when logstash is restarted :
root 61G 25G 36G 41% /

#2

Hello, please add your logstash configuration.

#3

Well, what's taking up all that space? du should give some clues.

#4

PFB my configurations :
input {

redis {
host => 'xx.xx.xxx.xxx'
port => 1234
password => 'passwd'
data_type => 'list'
key => 'key'
}
}

filter {

if("abcdsf" in [tags])
{

grok {
match => { "message" => "^=%{WORD:report_type} REPORT=+ (?%{MONTHDAY}-%{MONTH}-%{YEAR}::%{HOUR}:%{MINUTE}:%{SECOND}) ===.*$" }
}

multiline {
pattern => "(^=)"
negate => true
what => "previous"
}
}
else {
multiline {
pattern => "^%{TIMESTAMP_ISO8601}%{SPACE}%{NUMBER}?%{SPACE}?TRACE"
what => "previous"
}
grok {
match => { "message" => "%{TIMESTAMP_ISO8601:logdate}%{SPACE}%{NUMBER:pid}?%{SPACE}?(?AUDIT|CRITICAL|DEBUG|INFO|TRACE|WARNING|ERROR) [?\b%{NOTSPACE:mod}\b]?%{SPACE}?%{GREEDYDATA:logmessage}" }
}

}
}

output { stdout { codec => rubydebug } }

output {
elasticsearch {
hosts => ["elasticsearchEndpoint:443"]
ssl => true
keystore => "/etc/pki/Truststore.jks"
keystore_password => "passwd"
user => "passwd"
password => "passwd"
index => "logstash--%{+YYYY.MM.dd}"
}
stdout { codec => rubydebug }
}

#5

output { stdout { codec => rubydebug } }

You should remove the two stdout outputs. These cause all events passing through Logstash to be logged twice to Logstash's log file.

#6

Thanks for verifying the configs but how does the volume usage is reducing just by restarting logstash.

#7

Perhaps because the files that occupied the disk space were deleted while Logstash was running and kept them open, and once Logstash shuts down the disk space was reclaimed.

#8

As logstash keeps events in persistent queue .Is this contributing to the volume usage .Apart from this which files are kept opened by logstash?

#9

As logstash keeps events in persistent queue

What persistent queue are you talking about? Logstash has no built-in queue.

Is this contributing to the volume usage .Apart from this which files are kept opened by logstash?

Let's not spend time speculating. Find out what files are using up the space.

#10

I attended the elasticOn 2016 and this was the session which explained about persistent queues :
https://www.elastic.co/elasticon/conf/2016/sf/dive-deep-with-logstash-from-pipelines-to-persistent-queues

So, how can I find out which all files are contributing to disk usage in logstash ?

#11

ls -l /proc/{PID}/fd should tell you what files are open by process.

or

lsof -p {PID}

or

pfiles {PID}

But i think its the logstash log-output.

How do you start logstash? With --debug ?

#12

I attended the elasticOn 2016 and this was the session which explained about persistent queues :

I'm pretty sure the Logstash 2.2 pipeline change didn't actually include the persistent queue feature. Secondly, you're running Logstash 2.1 so none of that applies anyway.

So, how can I find out which all files are contributing to disk usage in logstash ?

As I said du would be a good start. @jupp gave a few other useful suggestions.

#13

I am running Logstash process in linux box where i configured DEBUG=1 in logstash.service file .

#14

Thanks magnus for confirming , i will test by disabling the debug and see if still the volumes are getting consumed . I configured DEBUG=1 in logstash.service file .

#15