when logstash enable queue.type: persisted, other parameters is default, i found logstash output very slow, but queue has buffered hundreds of megabytes of data. from visualize i got follow pic, normally data jitter frequency is not large.
I was looking at the same info. It doesn't make sense to me why I am Queuing so much when the output isn't that busy. To the point it just keeps growing. Anybody have a good best practice ?
[2018-03-08T09:40:44,689][DEBUG][logstash.runner ] -------- Logstash Settings (* means modified) ---------
[2018-03-08T09:40:44,689][DEBUG][logstash.runner ] node.name: "ent-mocsmlh01.us.ad.gannett.com."
[2018-03-08T09:40:44,689][DEBUG][logstash.runner ] *path.config: "/etc/logstash/conf.d"
[2018-03-08T09:40:44,689][DEBUG][logstash.runner ] *path.data: "/var/lib/logstash" (default: "/usr/share/logstash/data")
[2018-03-08T09:40:44,689][DEBUG][logstash.runner ] modules.cli: []
[2018-03-08T09:40:44,689][DEBUG][logstash.runner ] modules: []
[2018-03-08T09:40:44,689][DEBUG][logstash.runner ] modules_setup: false
[2018-03-08T09:40:44,690][DEBUG][logstash.runner ] config.test_and_exit: false
[2018-03-08T09:40:44,690][DEBUG][logstash.runner ] config.reload.automatic: false
[2018-03-08T09:40:44,690][DEBUG][logstash.runner ] config.reload.interval: 3000000000
[2018-03-08T09:40:44,693][DEBUG][logstash.runner ] config.support_escapes: false
[2018-03-08T09:40:44,694][DEBUG][logstash.runner ] metric.collect: true
[2018-03-08T09:40:44,694][DEBUG][logstash.runner ] pipeline.id: "main"
[2018-03-08T09:40:44,694][DEBUG][logstash.runner ] pipeline.system: false
[2018-03-08T09:40:44,694][DEBUG][logstash.runner ] *pipeline.workers: 6 (default: 2)
[2018-03-08T09:40:44,694][DEBUG][logstash.runner ] *pipeline.output.workers: 2 (default: 1)
[2018-03-08T09:40:44,694][DEBUG][logstash.runner ] pipeline.batch.size: 125
[2018-03-08T09:40:44,694][DEBUG][logstash.runner ] pipeline.batch.delay: 50
[2018-03-08T09:40:44,694][DEBUG][logstash.runner ] pipeline.unsafe_shutdown: false
[2018-03-08T09:40:44,694][DEBUG][logstash.runner ] pipeline.java_execution: false
[2018-03-08T09:40:44,694][DEBUG][logstash.runner ] pipeline.reloadable: true
[2018-03-08T09:40:44,694][DEBUG][logstash.runner ] path.plugins: []
[2018-03-08T09:40:44,694][DEBUG][logstash.runner ] config.debug: false
[2018-03-08T09:40:44,694][DEBUG][logstash.runner ] *log.level: "debug" (default: "info")
[2018-03-08T09:40:44,695][DEBUG][logstash.runner ] version: false
[2018-03-08T09:40:44,695][DEBUG][logstash.runner ] help: false
[2018-03-08T09:40:44,695][DEBUG][logstash.runner ] log.format: "plain"
[2018-03-08T09:40:44,695][DEBUG][logstash.runner ] http.host: "127.0.0.1"
[2018-03-08T09:40:44,695][DEBUG][logstash.runner ] http.port: 9600..9700
[2018-03-08T09:40:44,695][DEBUG][logstash.runner ] http.environment: "production"
[2018-03-08T09:40:44,695][DEBUG][logstash.runner ] *queue.type: "persisted" (default: "memory")
[2018-03-08T09:40:44,695][DEBUG][logstash.runner ] queue.drain: false
[2018-03-08T09:40:44,695][DEBUG][logstash.runner ] queue.page_capacity: 67108864
[2018-03-08T09:40:44,695][DEBUG][logstash.runner ] queue.max_bytes: 1073741824
[2018-03-08T09:40:44,695][DEBUG][logstash.runner ] *queue.max_events: 2048 (default: 0)
[2018-03-08T09:40:44,695][DEBUG][logstash.runner ] queue.checkpoint.acks: 1024
[2018-03-08T09:40:44,696][DEBUG][logstash.runner ] queue.checkpoint.writes: 1024
[2018-03-08T09:40:44,696][DEBUG][logstash.runner ] queue.checkpoint.interval: 1000
[2018-03-08T09:40:44,696][DEBUG][logstash.runner ] dead_letter_queue.enable: false
[2018-03-08T09:40:44,696][DEBUG][logstash.runner ] dead_letter_queue.max_bytes: 1073741824
[2018-03-08T09:40:44,696][DEBUG][logstash.runner ] slowlog.threshold.warn: -1
[2018-03-08T09:40:44,696][DEBUG][logstash.runner ] slowlog.threshold.info: -1
[2018-03-08T09:40:44,696][DEBUG][logstash.runner ] slowlog.threshold.debug: -1
[2018-03-08T09:40:44,696][DEBUG][logstash.runner ] slowlog.threshold.trace: -1
[2018-03-08T09:40:44,696][DEBUG][logstash.runner ] keystore.classname: "org.logstash.secret.store.backend.JavaKeyStore"
[2018-03-08T09:40:44,696][DEBUG][logstash.runner ] *keystore.file: "/etc/logstash/logstash.keystore" (default: "/usr/share/logstash/config/logstash.keystore")
[2018-03-08T09:40:44,696][DEBUG][logstash.runner ] *path.queue: "/var/lib/logstash/queue" (default: "/usr/share/logstash/data/queue")
[2018-03-08T09:40:44,696][DEBUG][logstash.runner ] *path.dead_letter_queue: "/var/lib/logstash/dead_letter_queue" (default: "/usr/share/logstash/data/dead_letter_queue")
[2018-03-08T09:40:44,697][DEBUG][logstash.runner ] *path.settings: "/etc/logstash" (default: "/usr/share/logstash/config")
[2018-03-08T09:40:44,697][DEBUG][logstash.runner ] *path.logs: "/var/log/logstash" (default: "/usr/share/logstash/logs")
[2018-03-08T09:40:44,697][DEBUG][logstash.runner ] --------------- Logstash Settings -------------------
I seem to have fixed some of my problem with tuning my http output re-try.
Switched back to memory before I added these but may have the same effect on persistent Queues. I was using the SumoLogic http output plugin and 3Gb of heap which ran out in <9 hrs.
Now my heap isn't growing above 20% used.
automatic_retries => 10
validate_after_inactivity => 3
request_timeout => 10
em..., i use elasticesearch output plugin, there seems to be no suitable option for this case.