Hi everyone,
We are using the logging pipeline Filebeat -> Logstash -> Dataset/Scalyr.
The issue I'm dealing with is that the Logstash memory keeps increasing until the OOM kills the pod (so we are talking about OS OOM kill, not the heap one).
With default settings, huge heap, low heap, low/high number of workers, and all combinations, the result is always the same.
The interesting thing is, that if I restrict the incoming traffic to a pod, the memory is still kept high. Even when the events are not there anymore and were successfully delivered to the target database.
Deployment details:
Logstash v8.13.0
Additional filters (latest versions):
logstash-filter-prune
logstash-filter-json_encode
logstash-output-scalyr
Kubernetes version v1.24.12
JVM min and max set to 50-75% of the physical memory
Mem: 30Gi
CPU: 14
Inputs:
Beats, TCP, Syslog
Filters:
Mutate, drop, truncate, json, grok
Output:
Scalyr
Java Native memory tracking output (The "Other" category is quite big)
jdk/bin/jcmd 1 VM.native_memory summary scale=mb
Defaulted container "logstash" out of: logstash, logstash-exporter
1:
Native Memory Tracking:
(Omitting categories weighting less than 1MB)
Total: reserved=28108MB, committed=26198MB
malloc: 9838MB #824166
mmap: reserved=18270MB, committed=16360MB
- Java Heap (reserved=15360MB, committed=15360MB)
(mmap: reserved=15360MB, committed=15360MB)
- Class (reserved=1032MB, committed=28MB)
(classes #25640)
( instance classes #24548, array classes #1092)
(malloc=8MB #125737) (peak=8MB #142766)
(mmap: reserved=1024MB, committed=21MB)
( Metadata: )
( reserved=128MB, committed=107MB)
( used=102MB)
( waste=4MB =3.91%)
( Class space:)
( reserved=1024MB, committed=21MB)
( used=17MB)
( waste=4MB =17.81%)
- Thread (reserved=868MB, committed=96MB)
(thread #862)
(stack: reserved=865MB, committed=93MB)
(malloc=1MB #5188) (peak=2MB #6050)
(arena=1MB #1722) (peak=63MB #1720)
- Code (reserved=247MB, committed=138MB)
(malloc=5MB #13466) (peak=11MB #29022)
(mmap: reserved=242MB, committed=133MB)
- GC (reserved=919MB, committed=919MB)
(malloc=284MB #84860) (peak=326MB #84115)
(mmap: reserved=635MB, committed=635MB)
- Compiler (reserved=11MB, committed=11MB)
(malloc=11MB #5957) (peak=11MB #5986)
- Internal (reserved=3MB, committed=3MB)
(malloc=3MB #54069) (peak=6MB #82163)
- Other (reserved=9465MB, committed=9465MB)
(malloc=9465MB #1358) (peak=9545MB #1361)
- Symbol (reserved=17MB, committed=17MB)
(malloc=15MB #505204) (peak=15MB #505205)
(arena=2MB #1) (at peak)
- Native Memory Tracking (reserved=13MB, committed=13MB)
(tracking overhead=13MB)
- Shared class space (reserved=16MB, committed=12MB)
(mmap: reserved=16MB, committed=12MB)
- Arena Chunk (reserved=9MB, committed=9MB)
(malloc=9MB #2061) (peak=333MB #9846)
- Tracing (reserved=15MB, committed=15MB)
(malloc=15MB #206) (at peak)
- Module (reserved=2MB, committed=2MB)
(malloc=2MB #9465) (at peak)
- Synchronization (reserved=1MB, committed=1MB)
(malloc=1MB #7670) (peak=1MB #7674)
- Metaspace (reserved=130MB, committed=109MB)
(malloc=2MB #3079) (at peak)
(mmap: reserved=128MB, committed=107MB)
Logstash config
config.reload.automatic: true
http.host: 0.0.0.0
http.port: 9600
log.format: json
path.config: /usr/share/logstash/pipeline
pipeline.batch.size: 2000
pipeline.ecs_compatibility: disabled
pipeline.workers: 30