No space left on device - Logstash

Elastic Stack Logstash
1

I have an ELK stack, hosted on a Linux, an EC2 VM running at AWS and my logstash works in the background on that same environment

I used to start logstash as a service using "systemctl start logstash" and it works normally

It starts to process the logs

The logs are .txt files, all stored at a directory that works like a file system, mounted on my S3 Bucket

So, I have my logstash .conf file, that has an input that look to this folder inside my EC2, using the file input plugin

And after like, 1 or 2 hours, logstash starts to log some errors like "No space left on device"

And the path next to this message is exactly the same path where the raw logs are stored inside the EC2

Like

Logs are stored at "/usr/share/logs/log1.txt"

And the error message was

No space left on device => [:path => " "/usr/share/logs/log1.txt"

So, i've stopped logstash using "systemctl stop logstash", waited like, 30 min and restarts it, and the logs starts to process again

I don't know why this is happening, and how can i prevent this from happening

Edit1: If I let logstash running, it stops to process everything when the "No space left..." errors starts to come out, i've made a test, waiting some time to logstash to stabilize, but nothing happens, anyway i have to stop logstash and restarts it

2

Are you actually running out of disk space?
What does your config look like?

3

When Logstash is not running, i'm not out of disk space, but when it starts to run, the disks starts do fill it up

My config file is like

input{
file{
    path => "/var/logs/*.txt"
    start_position => "beginning"
    codec => multiline {
        pattern => "timestamp="
        negate => true
        what => "previous"
    }
}
}
filter {
grok{ /*A SIMPLE PATTERN*/}
mutate(
    remove_field => {/*SOME CODE*/}
)
}
output {
elasticsearch{
    hosts => "MYHOST"
    index => "MYINDEX"
}
}

The conf file does his job, the logs actually process correctly, Elasticsearch index them, and they appear in Kibana

But after sometime, it's start to log some "No space left on device" messages, telling me that the path to the location where there are no space left, is the same location where the raw logs are stored

4

It'd be useful if you could share the log.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.