Dear Elastic,
I have some questions of Logstash to you.
Could you please answer my questions?
Best Regards
Robin
#####################
logstash server sepc
#####################
Cpu:4 core
Mem: 16g
Java heap: 8g
Logstash version: 6.2.x
NIC:10G
###########################
# logstash pipeline config
###########################
180817162922 root@test101 logstash # cat pipelines.yml |grep -i 'meraki' -A4
- pipeline.id: rsyslog-cisco-meraki
path.config: /etc/logstash/conf.d/rsyslog-cisco-meraki.conf
pipeline.batch.size: 2000
pipeline.workers: 4
queue.type: persisted
queue.max_bytes: 1024mb
###########################
# logstash pipeline status
###########################
180817162700 root@test101 logstash # curl -XGET '127.0.0.1:9600/_node/stats/pipelines/rsyslog-cisco-meraki?pretty'
{
..........
..........
..........
"queue" : {
"events" : 0,
"type" : "persisted",
"capacity" : {
"queue_size_in_bytes" : 99431,
"page_capacity_in_bytes" : 67108864,
"max_queue_size_in_bytes" : 1073741824,
"max_unread_events" : 0
},
"data" : {
"path" : "/data/logstash/queue/rsyslog-cisco-meraki",
"free_space_in_bytes" : 7171907584,
"storage_type" : "rootfs"
}
}
}
###############
# Questions
###############
1. what's the difference between pipeline.batch.size and queue.size ?
1). Is the pipeline.batch.size of pipeline input size, filter size or output size?
2). Is pipeline.batch.size the size to output to Elasticsarch at one time?
2.according to set up above, Are events processed by pipeline like this as below?
input-> queue(persisted) ->filer (take out events from queue ) -> output ES(Bulk requests)
3. How many events does logstash send to elasticsearch at one time(bulk requests) according to my setup above?
4. I found that there few options are removed since logstash 6 for elasticsearch output plugin. if we want to tweak the logsatash output rate to ES for throughput. how can we do that without the below options?
workers: 1 (by default) before
flush_size: 500 (by default) before
5. if I change persisted queue back to the memory-based queue, how many events would save in heap by default?