How to run multiple logstash instances on the same machine?

Elastic Stack Logstash
1

I want to run multiple logstash configurations on the same machine using different config files. As far as I know, if I put multiple config files in the /etc/logstash/conf.d/ directory, it will simply append the .conf files into one giant one which would cause unexpected behavior. Is there a clean way to have multiple instances running concurrently as a service?

I'm running Linux 14.04

Thanks for the help!

2

Hey @esamudio I recently dealt with running multiple logstash instances, and here's what I did. I use chef to add an additional logstash instance each time I need to add another applicaiton stack that requires an instance of logstash.

So I created a template in chef from the original logstash init.d script in /etc/init.d/ When a new instance of logstash is created, chef will stage a new init.d script with a few of the values swapped out:

LS_CONF_DIR=/etc/logstash/ls-someappname
name=ls-someappname

Then each instance of logstash has their respective config files staged under the directory their service is configured to point at:

/etc/logstash/ls-someappname/02-beats-input.conf
/etc/logstash/ls-someappname/10-filter.conf
/etc/logstash/ls-someappname/30-elasticsearch-output.conf

So your process will need to create a conf dir for each instance, and an init.d script for each instance. Then I can manage each service independently "sudo serivce ls-someappname status". I also assign each instance a different port number in the beats-input conf.

This is also nice, because you can update the filter.conf for one logstash instance, and restart it, without affecting the other instances. I dont know if this is the best way or what the accepted practice is for running multiple logstash instances, but I can tell you this works great.

2 Likes
3

I really appreciate the response. I'm currently using Ansible for some other stuff on my project so I think I should be able to incorporate what you're doing with chef using Ansible.

As for the ports, that only applies for overlapping services that use a given port, right? It makes sense if that's the case but just wanted to clarify.

Thanks again for the quick response!

4

Hey @esamudio you'd most likely need a unique port for each instance of logstash. I dont know if you can use the same port for multiple instances of logstash. You'd probably receive an error if ls-someapp1 is listening on 4501 when you try to start ls-someapp2 if its also configured in its filebeat input conf to listen on 4501.

5

I also tried two logstash instances. But only one instance was able to receive events. Can I how your logstash version?

6

Hey @yiju ,

I was able to have multiple instances of logstash running using version 2.x and also now on version 5. If you give more more information as to how you went about setting multiple instances, I should be able to help you

7

Hi @esamudio

I'm working with logstash 2.4 and filebeat. Filebeat sends the logs to two ports and I have two logstash instances each listen to one port, but only one of the instance is able to get the log at a time and the other one will lost the logs. It seems two instances are competing for the logs and the winner changes after some time.

8

Hi

I am new to ELK stack. Configured single instance of elasticsearch with a logstash and multiple filebeats.
I need to configure/set up a multiple connections for Logstash high availability - load balancer - Logstash cluster to handle failures. Is there a documentation or video that I could follow to do this setup?

9

Hi @yiju,

I seem to have misunderstood your problem. I can help you set up multiple, general instances of Logstash but I do not use filebeat so I cannot help you with that.

Good luck!