Not enough free disk space available to allocate persisted queue


(panaman) #1

I am testing out the new logstash 6.3.0 and all it does is error out and crash saying i don't have enough disk space if my queue.max_bytes: is set to anything higher than about 9400mb.

My data dir has 190GB free and I have tested this on different machines just to make sure i wasn't having some kind of hardware problem

bash-4.2$ cat config/logstash.yml | grep data
path.data: /usr/share/logstash/data

bash-4.2$ df -h /usr/share/logstash/data
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_root-lv_opt 190G 237M 190G 1% /usr/share/logstash/data

bash-4.2$ cat config/logstash.yml | grep queue
queue.checkpoint.acks: 1024
queue.checkpoint.writes: 1024
queue.max_bytes: 9500mb
queue.page_capacity: 1024mb
queue.type: persisted


(Dan Hermann) #2

@panaman, I am able to successfully configure and run on Linux with a persistent queue over 9400mb in size. On what OS are you attempting to run this?

As an aside, I would suggest that there are few cases where running with such a large PQ size is advantageous. Broadly speaking, there are three categories of Logstash pipelines:

  • Pipelines in which the inputs are faster than the outputs. In this scenario, a large PQ is generally not desirable because the outputs must generate back-pressure on the inputs to reduce the ingestion rate of events. Having a large PQ simply delays the point at which outputs can exert back-pressure on the inputs. A PQ size in the range of 100mb is probably sufficient for this scenario.
  • Pipelines in which the inputs are slower than the outputs. In this scenario, the queue depth will always be close to zero since the outputs process events faster than the inputs generate them and therefore a large PQ size is neither necessary nor useful. A PQ size of 100mb or less is usually sufficient for this scenario since it all has to accommodate is small event volumes due to occasional JVM jitter, GC pauses, etc.
  • Pipelines in which the inputs are usually slower than the outputs but occasional spikes in event volume are produced. This is the one scenario in which a larger PQ may be beneficial since it may allow your pipeline to absorb occasional spikes in event volume without exerting back-pressure on the inputs. If that is advantageous for your particular workload, you could size your PQ to accommodate those spikes.

Also, we've generally seen the best performance with a queue.page_capacity of 64mb. With a page size of 1024mb, I would expect you to experience degraded performance in a number of situations such as at Logstash startup time.


(panaman) #3

we process a high volume of logs (16 logstash servers & 36 workers each) and when things go south we sometimes get 300gb of logs backed up in a few minutes so having the large Q is needed.

I have not narrowed down the exact size of the Q where it fails, but i can definitely say if i put "10gb" as my Q size it will say that I don't have enough space. If I put "9gb" it works fine.
I adjusted my settings so the max_bytes is "10gb" and the page_capacity is "64mb" and it crashes and still saying that i don't have enough disk space.

I am using the docker image and i use a bind mount to the host disk


(panaman) #4

ok, so I think i might have found something... when doing a df -h inside the docker container this might explain why it works at 9gb and not 10gb. I believe the check is being applied to the root filesystem and not the path.data

bash-4.2$ df -h
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/docker-253:3-16797764-17c0cf8047810bbebc318f25609c782743db3ffddf19d5efcd2e654254cd3b8b 10G 808M 9.3G 8% /
tmpfs 64M 0 64M 0% /dev
tmpfs 32G 0 32G 0% /sys/fs/cgroup
/dev/mapper/vg_root-lv_docker 16G 1.4G 15G 9% /etc/hosts
shm 64M 0 64M 0% /dev/shm
/dev/mapper/vg_root-lv_log 8.0G 895M 7.2G 11% /usr/share/logstash/logs
/dev/mapper/vg_root-lv_opt 175G 79M 175G 1% /usr/share/logstash/data
tmpfs 32G 0 32G 0% /proc/scsi
tmpfs 32G 0 32G 0% /sys/firmware


(Dan Hermann) #5

@panaman, you're right about the check for free disk space. I've opened an issue (https://github.com/elastic/logstash/issues/9766) to address that problem.


(panaman) #6

module LogStash

In the event that we're requiring this file without bootstrap/environment.rb

if !defined?(LogStash::Environment::LOGSTASH_HOME)
module Environment
LOGSTASH_HOME = Stud::Temporary.directory("logstash-home")
Dir.mkdir(::File.join(LOGSTASH_HOME, "data"))
end
end

[
Setting::String.new("node.name", Socket.gethostname),
Setting::NullableString.new("path.config", nil, false),
Setting::WritableDirectory.new("path.data", ::File.join(LogStash::Environment::LOGSTASH_HOME, "data")),


(Dan Hermann) #7

This bug has been fixed in https://github.com/elastic/logstash/pull/9767 and is scheduled for the next patch release.


(panaman) #8

thanks! i guess i will just wait to upgrade when 6.3.1 is released


(system) #9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.