when logstash enable queue.type: persisted, other parameters is default, i found logstash output very slow, but queue has buffered hundreds of megabytes of data. from visualize i got follow pic, normally data jitter frequency is not large.
I was looking at the same info. It doesn't make sense to me why I am Queuing so much when the output isn't that busy. To the point it just keeps growing. Anybody have a good best practice ?
[2018-03-08T09:40:44,689][DEBUG][logstash.runner ] -------- Logstash Settings (* means modified) ---------
[2018-03-08T09:40:44,689][DEBUG][logstash.runner ] node.name: "ent-mocsmlh01.us.ad.gannett.com."
[2018-03-08T09:40:44,689][DEBUG][logstash.runner ] *path.config: "/etc/logstash/conf.d"
[2018-03-08T09:40:44,689][DEBUG][logstash.runner ] *path.data: "/var/lib/logstash" (default: "/usr/share/logstash/data")
[2018-03-08T09:40:44,689][DEBUG][logstash.runner ] modules.cli: []
[2018-03-08T09:40:44,689][DEBUG][logstash.runner ] modules: []
[2018-03-08T09:40:44,689][DEBUG][logstash.runner ] modules_setup: false
[2018-03-08T09:40:44,690][DEBUG][logstash.runner ] config.test_and_exit: false
[2018-03-08T09:40:44,690][DEBUG][logstash.runner ] config.reload.automatic: false
[2018-03-08T09:40:44,690][DEBUG][logstash.runner ] config.reload.interval: 3000000000
[2018-03-08T09:40:44,693][DEBUG][logstash.runner ] config.support_escapes: false
[2018-03-08T09:40:44,694][DEBUG][logstash.runner ] metric.collect: true
[2018-03-08T09:40:44,694][DEBUG][logstash.runner ] pipeline.id: "main"
[2018-03-08T09:40:44,694][DEBUG][logstash.runner ] pipeline.system: false
[2018-03-08T09:40:44,694][DEBUG][logstash.runner ] *pipeline.workers: 6 (default: 2)
[2018-03-08T09:40:44,694][DEBUG][logstash.runner ] *pipeline.output.workers: 2 (default: 1)
[2018-03-08T09:40:44,694][DEBUG][logstash.runner ] pipeline.batch.size: 125
[2018-03-08T09:40:44,694][DEBUG][logstash.runner ] pipeline.batch.delay: 50
[2018-03-08T09:40:44,694][DEBUG][logstash.runner ] pipeline.unsafe_shutdown: false
[2018-03-08T09:40:44,694][DEBUG][logstash.runner ] pipeline.java_execution: false
[2018-03-08T09:40:44,694][DEBUG][logstash.runner ] pipeline.reloadable: true
[2018-03-08T09:40:44,694][DEBUG][logstash.runner ] path.plugins: []
[2018-03-08T09:40:44,694][DEBUG][logstash.runner ] config.debug: false
[2018-03-08T09:40:44,694][DEBUG][logstash.runner ] *log.level: "debug" (default: "info")
[2018-03-08T09:40:44,695][DEBUG][logstash.runner ] version: false
[2018-03-08T09:40:44,695][DEBUG][logstash.runner ] help: false
[2018-03-08T09:40:44,695][DEBUG][logstash.runner ] log.format: "plain"
[2018-03-08T09:40:44,695][DEBUG][logstash.runner ] http.host: "127.0.0.1"
[2018-03-08T09:40:44,695][DEBUG][logstash.runner ] http.port: 9600..9700
[2018-03-08T09:40:44,695][DEBUG][logstash.runner ] http.environment: "production"
[2018-03-08T09:40:44,695][DEBUG][logstash.runner ] *queue.type: "persisted" (default: "memory")
[2018-03-08T09:40:44,695][DEBUG][logstash.runner ] queue.drain: false
[2018-03-08T09:40:44,695][DEBUG][logstash.runner ] queue.page_capacity: 67108864
[2018-03-08T09:40:44,695][DEBUG][logstash.runner ] queue.max_bytes: 1073741824
[2018-03-08T09:40:44,695][DEBUG][logstash.runner ] *queue.max_events: 2048 (default: 0)
[2018-03-08T09:40:44,695][DEBUG][logstash.runner ] queue.checkpoint.acks: 1024
[2018-03-08T09:40:44,696][DEBUG][logstash.runner ] queue.checkpoint.writes: 1024
[2018-03-08T09:40:44,696][DEBUG][logstash.runner ] queue.checkpoint.interval: 1000
[2018-03-08T09:40:44,696][DEBUG][logstash.runner ] dead_letter_queue.enable: false
[2018-03-08T09:40:44,696][DEBUG][logstash.runner ] dead_letter_queue.max_bytes: 1073741824
[2018-03-08T09:40:44,696][DEBUG][logstash.runner ] slowlog.threshold.warn: -1
[2018-03-08T09:40:44,696][DEBUG][logstash.runner ] slowlog.threshold.info: -1
[2018-03-08T09:40:44,696][DEBUG][logstash.runner ] slowlog.threshold.debug: -1
[2018-03-08T09:40:44,696][DEBUG][logstash.runner ] slowlog.threshold.trace: -1
[2018-03-08T09:40:44,696][DEBUG][logstash.runner ] keystore.classname: "org.logstash.secret.store.backend.JavaKeyStore"
[2018-03-08T09:40:44,696][DEBUG][logstash.runner ] *keystore.file: "/etc/logstash/logstash.keystore" (default: "/usr/share/logstash/config/logstash.keystore")
[2018-03-08T09:40:44,696][DEBUG][logstash.runner ] *path.queue: "/var/lib/logstash/queue" (default: "/usr/share/logstash/data/queue")
[2018-03-08T09:40:44,696][DEBUG][logstash.runner ] *path.dead_letter_queue: "/var/lib/logstash/dead_letter_queue" (default: "/usr/share/logstash/data/dead_letter_queue")
[2018-03-08T09:40:44,697][DEBUG][logstash.runner ] *path.settings: "/etc/logstash" (default: "/usr/share/logstash/config")
[2018-03-08T09:40:44,697][DEBUG][logstash.runner ] *path.logs: "/var/log/logstash" (default: "/usr/share/logstash/logs")
[2018-03-08T09:40:44,697][DEBUG][logstash.runner ] --------------- Logstash Settings -------------------
1 Like
I seem to have fixed some of my problem with tuning my http output re-try.
Switched back to memory before I added these but may have the same effect on persistent Queues. I was using the SumoLogic http output plugin and 3Gb of heap which ran out in <9 hrs.
Now my heap isn't growing above 20% used.
automatic_retries => 10
validate_after_inactivity => 3
request_timeout => 10
em..., i use elasticesearch output plugin, there seems to be no suitable option for this case.
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.