# Clarification towards logstash batch\_size and memory usage **URL:** \ **Category:** Logstash\ **Created:** [September 1, 2020, 10:09am UTC](https://discuss.elastic.co/t/clarification-towards-logstash-batch-size-and-memory-usage/247073 "2020-09-01T10:09:17Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![vaishno.avi](https://avatars.discourse-cdn.com/v4/letter/v/f0a364/32.png) [@vaishno.avi](https://discuss.elastic.co/u/vaishno.avi)\ **Post date:** [September 1, 2020, 10:09am UTC](https://discuss.elastic.co/t/clarification-towards-logstash-batch-size-and-memory-usage/247073/1 "2020-09-01T10:09:17Z")
Hello All, I have two concerns 1. I have a pipeline.yml where I am explicitly setting batch size as 100, but when I start logstash it shows me 125, is this an expected behavior ? config > - pipeline.id: filebeat > path.config: "/opt/logstash/config/filebeat.conf" > pipeline.workers: 1 > pipeline.batch.size: 100 > queue.type: persisted > path.queue: /opt/logstash/data/queue > queue.max\_events: 0 > queue.max\_bytes: 4096mb Logstash stats : > curl -XGET 'localhost:9600/\_node/stats/pipelines/filebeat?pretty' > { > "host" : "xxxx", > "version" : "7.5.0", > "http\_address" : "127.0.0.1:9600", > "id" : "16bb4a4c-1b21-4ffb-ad26-439eeb84446a", > "name" : "xxxxx", > "ephemeral\_id" : "e51c630a-761d-4f4e-b074-4866568fadc3", > "status" : "green", > "snapshot" : false, > "pipeline" : { > "workers" : 2, > "batch\_size" : 125, > "batch\_delay" : 50 > }, > "pipelines" : { > "filebeat" : { > "events" : { > "queue\_push\_duration\_in\_millis" : 8989, > "duration\_in\_millis" : 74097, > "out" : 37201, > "filtered" : 37201, > "in" : 81258 > }, 1. Where does logstash uses the rest of the memory - I am using cadvisor to monitor logstash memory - logstash is running with 4gb jvm memory > ## JVM configuration > > # Xms represents the initial size of total heap space > # Xmx represents the maximum size of total heap space > -Xms4g > -Xmx4g - Logstash stats and docker stats show very less memory consumption, but cadvisor shows a lot. I am using `queue.type: persisted`, so events should not consume memory - I suppose that there would be non-heap memory as well, but that accounts to 1/2 gb over a long period of time. > curl -XGET 'localhost:9600/\_node/stats/jvm?pretty' > { > "host" : "xxxxx", > "version" : "7.5.0", > "http\_address" : "127.0.0.1:9600", > "id" : "16bb4a4c-1b21-4ffb-ad26-439eeb84446a", > "name" : "xxxxx", > "ephemeral\_id" : "e51c630a-761d-4f4e-b074-4866568fadc3", > "status" : "green", > "snapshot" : false, > "pipeline" : { > "workers" : 2, > "batch\_size" : 125, > "batch\_delay" : 50 > }, > "jvm" : { > "threads" : { > "count" : 39, > "peak\_count" : 43 > }, > "mem" : { > "heap\_used\_percent" : 6, > "heap\_committed\_in\_bytes" : 4277534720, > "heap\_max\_in\_bytes" : 4277534720, > "heap\_used\_in\_bytes" : 298714216, > "non\_heap\_used\_in\_bytes" : 184141792, > "non\_heap\_committed\_in\_bytes" : 209522688, > "pools" : { > "young" : { > "peak\_used\_in\_bytes" : 139591680, > "committed\_in\_bytes" : 139591680, > "peak\_max\_in\_bytes" : 139591680, > "max\_in\_bytes" : 139591680, > "used\_in\_bytes" : 68886720 > }, > "survivor" : { > "peak\_used\_in\_bytes" : 17432576, > "committed\_in\_bytes" : 17432576, > "peak\_max\_in\_bytes" : 17432576, > "max\_in\_bytes" : 17432576, > "used\_in\_bytes" : 1483960 > }, > "old" : { > "peak\_used\_in\_bytes" : 228343536, > "committed\_in\_bytes" : 4120510464, > "peak\_max\_in\_bytes" : 4120510464, > "max\_in\_bytes" : 4120510464, > "used\_in\_bytes" : 228343536 > } > } > }, > "gc" : { > "collectors" : { > "young" : { > "collection\_time\_in\_millis" : 12783, > "collection\_count" : 340 > }, > "old" : { > "collection\_time\_in\_millis" : 811, > "collection\_count" : 3 > } > } > }, > "uptime\_in\_millis" : 557542 > } > } docker stats : > docker stats xxxxx --no-stream > CONTAINER ID NAME CPU % MEM USAGE / LIMIT MEM % NET I/O BLOCK I/O PIDS > 8b12fd8ee990 xxxxx 28.90% 793.6MiB / 11.56GiB 6.71% 159MB / 52.3MB 187MB / 2.78GB 51 Cadvisor : Query used `container_memory_working_set_bytes` ![image](https://us1.discourse-cdn.com/elastic/original/3X/f/2/f2c5bc9f013ab0d83b8a745f84841007bb9250d3.png) Regards Ashish ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [September 1, 2020, 3:41pm UTC](https://discuss.elastic.co/t/clarification-towards-logstash-batch-size-and-memory-usage/247073/2 "2020-09-01T15:41:20Z")
> [@vaishno.avi](#): > > I have a pipeline.yml where I am explicitly setting batch size as 100, but when I start logstash it shows me 125, is this an expected behavior ? No. I suggest you check the indentation in your pipelines.yml (it's plural, right?) to make sure those settings apply to the pipeline id you want them to. ---
**Author:** ![vaishno.avi](https://avatars.discourse-cdn.com/v4/letter/v/f0a364/32.png) [@vaishno.avi](https://discuss.elastic.co/u/vaishno.avi)\ **Post date:** [September 2, 2020, 6:54am UTC](https://discuss.elastic.co/t/clarification-towards-logstash-batch-size-and-memory-usage/247073/3 "2020-09-02T06:54:18Z")
> [@Badger](#): > > I suggest you check the indentation in y @Badger : The file name is pipelines.yml (Sorry for the typo above) Here is the config ![image](https://us1.discourse-cdn.com/elastic/original/3X/6/1/61c1476bc84e6bedf5d24346259c62ed456ee501.png) Regards Ashish ---
**Author:** ![vaishno.avi](https://avatars.discourse-cdn.com/v4/letter/v/f0a364/32.png) [@vaishno.avi](https://discuss.elastic.co/u/vaishno.avi)\ **Post date:** [September 3, 2020, 1:37pm UTC](https://discuss.elastic.co/t/clarification-towards-logstash-batch-size-and-memory-usage/247073/4 "2020-09-03T13:37:37Z")
@Badger : Did you see my comments? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [September 3, 2020, 3:37pm UTC](https://discuss.elastic.co/t/clarification-towards-logstash-batch-size-and-memory-usage/247073/5 "2020-09-03T15:37:02Z")
Yes, and I cannot explain why that would be happening. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [October 1, 2020, 3:37pm UTC](https://discuss.elastic.co/t/clarification-towards-logstash-batch-size-and-memory-usage/247073/6 "2020-10-01T15:37:13Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.