Logstash persistent queue settings

Elastic Stack Logstash
1

Using logstash 7.13
We are running logstash on linux server which has 4 cpu(s) 15gb of ram of 100gb of ssd. We are using persistent queue settings in logstash but we actually can't sure is this right configuration or can it be improve.
logstash.yml file

# ------------ Queuing Settings --------------
#
# Internal queuing model, "memory" for legacy in-memory based queuing and
# "persisted" for disk-based acked queueing. Defaults is memory
#
queue.type: persisted
#
# If using queue.type: persisted, the directory path where the data files will be stored.
# Default is path.data/queue
#
path.queue: /var/lib/logstash/data_queue
#
# If using queue.type: persisted, the page data files size. The queue data consists of
# append-only data files separated into pages. Default is 64mb
#
#queue.page_capacity: 64mb
#
# If using queue.type: persisted, the maximum number of unread events in the queue.
# Default is 0 (unlimited)
#
# queue.max_events: 0
#
# If using queue.type: persisted, the total capacity of the queue in number of bytes.
# If you would like more unacked events to be buffered in Logstash, you can increase the
# capacity using this setting. Please make sure your disk drive has capacity greater than
# the size specified here. If both max_bytes and max_events are specified, Logstash will pick
# whichever criteria is reached first
# Default is 1024mb or 1gb
#
#queue.max_bytes: 1024mb
#
# If using queue.type: persisted, the maximum number of acked events before forcing a checkpoint
# Default is 1024, 0 for unlimited
#
# queue.checkpoint.acks: 1024
#
# If using queue.type: persisted, the maximum number of written events before forcing a checkpoint
# Default is 1024, 0 for unlimited
#
# queue.checkpoint.writes: 1024
#
# If using queue.type: persisted, the interval in milliseconds when a checkpoint is forced on the head page
# Default is 1000, 0 for no periodic checkpoint.
#
# queue.checkpoint.interval: 1000
#

pipeline.yml

[root@logstash-helk ~]# cat /etc/logstash/pipelines.yml
# This file is where you define your pipelines. You can define multiple.
# For more information on multiple pipelines, see the documentation:
#   https://www.elastic.co/guide/en/logstash/current/multiple-pipelines.html


- pipeline.id: winlogbeat
  #pipeline.batch.size: 2000
  #pipeline.workers: 6
  path.config: "/usr/share/logstash/pipeline/winlogbeat/"
  queue.type: persisted

- pipeline.id: packetbeat
  #pipeline.batch.size: 2000
  #pipeline.workers: 6
  path.config: "/usr/share/logstash/pipeline/packetbeat/"
  queue.type: persisted

- pipeline.id: pfsense
  #pipeline.batch.size: 2000
  #pipeline.workers: 6
  path.config: "/usr/share/logstash/pipeline/pfsense/"
  queue.type: persisted


- pipeline.id: vpn
   #pipeline.batch.size: 2000
    #    #pipeline.workers: 6
  path.config: "/usr/share/logstash/pipeline/vpn/"
  queue.type: persisted

Basically we want to use persistent queue what configuration do we need to configure.We are using by default configutration.We dont know page capacity.

2

Did you read Persistent queues (PQ) | Logstash Reference [7.14] | Elastic?

3

yes but we are using by default settings we just enable queue.type: persisted.

4

Then I'd just use the defaults, unless there are other issues that you are seeing.

5

ya i agree with you..but we dont know what value can we provide in queue.page_capacity.

6

Well, from the docs, emphasis mine;

  • queue.page_capacity : The maximum size of a queue page in bytes. The queue data consists of append-only files called "pages". The default size is 64mb. Changing this value is unlikely to have performance benefits.
7

we didn't change this.Can we change this or upto what values it can be change

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.