I am testing out the new logstash 6.3.0 and all it does is error out and crash saying i don't have enough disk space if my queue.max_bytes: is set to anything higher than about 9400mb.
My data dir has 190GB free and I have tested this on different machines just to make sure i wasn't having some kind of hardware problem
bash-4.2$ cat config/logstash.yml | grep data
path.data: /usr/share/logstash/data
bash-4.2$ df -h /usr/share/logstash/data
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/vg_root-lv_opt 190G 237M 190G 1% /usr/share/logstash/data
@panaman, I am able to successfully configure and run on Linux with a persistent queue over 9400mb in size. On what OS are you attempting to run this?
As an aside, I would suggest that there are few cases where running with such a large PQ size is advantageous. Broadly speaking, there are three categories of Logstash pipelines:
Pipelines in which the inputs are faster than the outputs. In this scenario, a large PQ is generally not desirable because the outputs must generate back-pressure on the inputs to reduce the ingestion rate of events. Having a large PQ simply delays the point at which outputs can exert back-pressure on the inputs. A PQ size in the range of 100mb is probably sufficient for this scenario.
Pipelines in which the inputs are slower than the outputs. In this scenario, the queue depth will always be close to zero since the outputs process events faster than the inputs generate them and therefore a large PQ size is neither necessary nor useful. A PQ size of 100mb or less is usually sufficient for this scenario since it all has to accommodate is small event volumes due to occasional JVM jitter, GC pauses, etc.
Pipelines in which the inputs are usually slower than the outputs but occasional spikes in event volume are produced. This is the one scenario in which a larger PQ may be beneficial since it may allow your pipeline to absorb occasional spikes in event volume without exerting back-pressure on the inputs. If that is advantageous for your particular workload, you could size your PQ to accommodate those spikes.
Also, we've generally seen the best performance with a queue.page_capacity of 64mb. With a page size of 1024mb, I would expect you to experience degraded performance in a number of situations such as at Logstash startup time.
we process a high volume of logs (16 logstash servers & 36 workers each) and when things go south we sometimes get 300gb of logs backed up in a few minutes so having the large Q is needed.
I have not narrowed down the exact size of the Q where it fails, but i can definitely say if i put "10gb" as my Q size it will say that I don't have enough space. If I put "9gb" it works fine.
I adjusted my settings so the max_bytes is "10gb" and the page_capacity is "64mb" and it crashes and still saying that i don't have enough disk space.
I am using the docker image and i use a bind mount to the host disk
ok, so I think i might have found something... when doing a df -h inside the docker container this might explain why it works at 9gb and not 10gb. I believe the check is being applied to the root filesystem and not the path.data
In the event that we're requiring this file without bootstrap/environment.rb
if !defined?(LogStash::Environment::LOGSTASH_HOME)
module Environment
LOGSTASH_HOME = Stud::Temporary.directory("logstash-home")
Dir.mkdir(::File.join(LOGSTASH_HOME, "data"))
end
end
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.