Logstash 2.1 is filling my Disk volumes

Elastic Stack Logstash
1

Hi Team , I am facing an issue that when my logstash process is running for 4-5 days , my disk volume on the machine where logstash is installed is getting exhausted and i have to restart the logstash process to free the volume . Can anyone please let me know how logstash exhausts the volumes on the machine.

Volume when logstash was running :slight_smile:

root 61G 61G 32K 100% /

Volume when logstash is restarted :
root 61G 25G 36G 41% /

1 Like
2

Hello, please add your logstash configuration.

3

Well, what's taking up all that space? du should give some clues.

4

PFB my configurations :
input {

redis {
host => 'xx.xx.xxx.xxx'
port => 1234
password => 'passwd'
data_type => 'list'
key => 'key'
}
}

filter {

if("abcdsf" in [tags])
{

grok {
match => { "message" => "^=%{WORD:report_type} REPORT=+ (?%{MONTHDAY}-%{MONTH}-%{YEAR}::%{HOUR}:%{MINUTE}:%{SECOND}) ===.*$" }
}

multiline {
pattern => "(^=)"
negate => true
what => "previous"
}
}
else {
multiline {
pattern => "^%{TIMESTAMP_ISO8601}%{SPACE}%{NUMBER}?%{SPACE}?TRACE"
what => "previous"
}
grok {
match => { "message" => "%{TIMESTAMP_ISO8601:logdate}%{SPACE}%{NUMBER:pid}?%{SPACE}?(?AUDIT|CRITICAL|DEBUG|INFO|TRACE|WARNING|ERROR) [?\b%{NOTSPACE:mod}\b]?%{SPACE}?%{GREEDYDATA:logmessage}" }
}

}
}

output { stdout { codec => rubydebug } }

output {
elasticsearch {
hosts => ["elasticsearchEndpoint:443"]
ssl => true
keystore => "/etc/pki/Truststore.jks"
keystore_password => "passwd"
user => "passwd"
password => "passwd"
index => "logstash--%{+YYYY.MM.dd}"
}
stdout { codec => rubydebug }
}

5

output { stdout { codec => rubydebug } }

You should remove the two stdout outputs. These cause all events passing through Logstash to be logged twice to Logstash's log file.

1 Like
6

Thanks for verifying the configs but how does the volume usage is reducing just by restarting logstash.

7

Perhaps because the files that occupied the disk space were deleted while Logstash was running and kept them open, and once Logstash shuts down the disk space was reclaimed.

8

As logstash keeps events in persistent queue .Is this contributing to the volume usage .Apart from this which files are kept opened by logstash?

9

As logstash keeps events in persistent queue

What persistent queue are you talking about? Logstash has no built-in queue.

Is this contributing to the volume usage .Apart from this which files are kept opened by logstash?

Let's not spend time speculating. Find out what files are using up the space.

10

I attended the elasticOn 2016 and this was the session which explained about persistent queues :
https://www.elastic.co/elasticon/conf/2016/sf/dive-deep-with-logstash-from-pipelines-to-persistent-queues

So, how can I find out which all files are contributing to disk usage in logstash ?

11

ls -l /proc/{PID}/fd should tell you what files are open by process.

or

lsof -p {PID}

or

pfiles {PID}

But i think its the logstash log-output.

How do you start logstash? With --debug ?

12

I attended the elasticOn 2016 and this was the session which explained about persistent queues :

I'm pretty sure the Logstash 2.2 pipeline change didn't actually include the persistent queue feature. Secondly, you're running Logstash 2.1 so none of that applies anyway.

So, how can I find out which all files are contributing to disk usage in logstash ?

As I said du would be a good start. @jupp gave a few other useful suggestions.

13

I am running Logstash process in linux box where i configured DEBUG=1 in logstash.service file .

14

Thanks magnus for confirming , i will test by disabling the debug and see if still the volumes are getting consumed . I configured DEBUG=1 in logstash.service file .