Beginners ELK design doubts

Elastic Stack Logstash
1

Hi
I am exploring the ELK framework for creating a centralized logging system across all our application(java,ruby,php) running on ec2 ubuntu servers . After going through some readings I have few doubts :

  1. How Is log stash indexer scalable horizontally ? Can we simply configure it behind a load balancer and simple keep adding/removing boxes in variance with load ? Is it stateless ?

  2. How to decide whether to use kafka queue or redis queue ?

3)Will I need to correct logging formats in all my applications for it to be useful or that can be done by logstash itself ? Can single logstash server handle logs with different formats from different applications ?

  1. How is filebeat in comparison to logstash forwarder ?
2
  1. Yes to those.
  2. What's easier?
  3. That's exactly what LS is for!
  4. Don't use the latter, it's 100% unsupported.
1 Like
3

Thanks Mark for quick response.

Regarding Kafka vs Redis , what factors should help in deciding ?
Factors I would consider is :
a) fault tolerance
b) cost
c) ease of maintenance/setup

As per my understanding, a) and b) would be in favor of kafka while c) in favor redis

Would be good to have other opinions as well

4

That's a fair summary of the two, based on my experience.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.