Using logstash 8.18.14
We are running logstash on Dev and QA environments with the same version and we observed JVM usage of almost 100%.
We process approximately 3000 logs/sec and of size ~1.5KB. Here are the logstash settings. Could you help in checking this issue.
Please let me if what more details needed from my end.
logstash.yml: |
http.host: 0.0.0.0
pipeline.batch.size: 250
pipeline.workers: 6
xpack.monitoring.enabled: false
pipeline.batch.delay: 30
dead_letter_queue.max_bytes: 2147483648
# Add queue settings for better memory management
queue.type: persisted
queue.max_bytes: 2gb
logstashJavaOpts: "-Xmx8g -Xms8g -XX:+UseG1GC -XX:MaxGCPauseMillis=200 -XX:+UseStringDeduplication -XX:G1HeapRegionSize=8m -XX:InitiatingHeapOccupancyPercent=35 -XX:+UnlockExperimentalVMOptions -XX:+DisableExplicitGC"
replicas: 4
resources:
requests:
cpu: "6"
memory: "16Gi"
limits:
cpu: "10"
memory: "20Gi"
Logstash Image Dockerfile content
FROM docker.elastic.co/logstash/logstash:8.18.4
RUN bin/logstash-plugin install logstash-filter-age
RUN bin/logstash-plugin install logstash-filter-http
RUN bin/logstash-plugin install logstash-input-google_pubsub
USER root
RUN apt-get update
RUN apt-get install -y shared-mime-info
USER logstash
RUN bin/logstash-plugin install logstash-input-google_cloud_storage