Hi Elastic team,
Currently, I'm using many Logstash instances to parse the customer logs. There are around 8 instances / server with 4GB JVM settings (-Xmx 4GB -Xms 4GB). The server have CPU 16 and Memory for 64GB.
In this situation, I found these issues.
- If a filter broke, it will broke for all customers.
- If a filter/grok need to update, it will impacts to all customers (restart)
- The flooding logs always take priority.
I would like to enhance the pipeline by customer separation.
customer1..N --> Logstash --> ES
customer1 --> Logstash_cust1 --> ES
customer2 --> Logstash_cust2 --> ES
customerN --> Logstash_custN --> ES
I think the Logstash docker can solve this issue. However, do you have the best practice or guide lines for container configuration?
- JVM for each container or should I limit the amount of the memory at the docker?
- The number of Logstash worker per container. In case the physcial have CPU 16 RAM 64GB, and we have 100 customers. Should I set only 1 worker for container?