JVM settings for Logstash container

Elastic Stack Logstash
1

Hi Elastic team,

Currently, I'm using many Logstash instances to parse the customer logs. There are around 8 instances / server with 4GB JVM settings (-Xmx 4GB -Xms 4GB). The server have CPU 16 and Memory for 64GB.

In this situation, I found these issues.

  • If a filter broke, it will broke for all customers.
  • If a filter/grok need to update, it will impacts to all customers (restart)
  • The flooding logs always take priority.

I would like to enhance the pipeline by customer separation.

From
customer1..N --> Logstash --> ES

To
customer1 --> Logstash_cust1 --> ES
customer2 --> Logstash_cust2 --> ES
...
customerN --> Logstash_custN --> ES

I think the Logstash docker can solve this issue. However, do you have the best practice or guide lines for container configuration?

  • JVM for each container or should I limit the amount of the memory at the docker?
  • The number of Logstash worker per container. In case the physcial have CPU 16 RAM 64GB, and we have 100 customers. Should I set only 1 worker for container?

Thank you,
Worapoj

2

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.