Multiple Logstash Instances on Single Server

Elastic Stack Logstash
1

Hi,

I would like to have several distinct logstash instances running on the same server. What is the best way to accomplish this?

I already have a single instance of logstash running, it monitors a large number of files and folders and is performing well.I now have a situation where I need to run a completely different configuration file to perform entirely different activities on a completely different set of folders and files on the same server.

When I try to run the second instance I get this error:

Error: Exception thrown by the agent : java.rmi.server.ExportException: Port already in use: 9010; nested exception is:
        java.net.BindException: Address already in use: JVM_Bind

I very much hope that I don't need to extend the paths array on the existing (working) logstash config file and add complexity there to get it to differentiate between inputs all in the same config file. That will get very messy.

Is there a good way to run two or more logstash instances on one machine?

Thanks.

2

It's totally possible to run multiple instances of Logstash, but they shouldn't share any resources. In your case, both instance must not attempt to listen on port 9010.

Assuming you're running something Linux-like you'll want to make a copy of /etc/init.d/logstash and adjust the name variable near the top of the file. That'll cause the init script to source different files from /etc/default or /etc/sysconfig which gives you an opportunity to override some of the settings (like LS_CONF_DIR).

3

Tangental question to this, what's the cited benefit for having multiple logstash agents vs. a single logstash agent handling multiple inputs? In what scenarios would this be appropriate? Is there a best practice?

I've noticed that most logstash-using infrastructures often cram everything into the conf.d folder and run it as a single service. Personally I set it up where each input is a separate service on a separate logstash agent? Is either preferrable/advantageous?

My reasoning for separating it out into services for each pipeline was for easier control over re-ingestion during development. For instance, I did not want to stop service for logstash and have it affect every single input that I had going to each output.

If I shutdown one service that is only tied to a single input, I'm only affecting that one input vs. all of them. I see this as a huge benefit but I rarely see this being done in publicly visible projects... am I missing something?

4

I don't think there's a definitive best practice here. If multiple instances work for you, great. I don't believe I've felt a need for shutting down just one input.

5

Hi Magnus

Sorry for resurrecting such an old thread. Thanks for your input but I am using a Windows machine for this and hoped you, or someone, could elaborate on which settings/files need to be updated to avoid this error on Windows? Using Logstash 1.5

Thanks!

6

This can be resolved on Windows machines by adjusting the values in the startup.bat file.
Not that you will find any documentation or support about that anywhere :rage:
Bit like trying to install and run locally developed plugins.... :rage:

7

I just run them seperately and put into background. for me, it is easier to have configuration in a single file, even a couple of hundred lines long, rather than many if then else to handle different input types.

it is easier to control, make modification and restart ingestion as mentioned at the beginning. Not to mentioned in some cases, plugins (especially community maintained) may have memory leak which can crash the logstash.

the main benefits of running as a service is in production environment, you can make sure logstash is running even after reboot. If you are in that stage and have limited number of input, then I guess put everything in to a single directory is manageable.

I just do not like the behavior of concatenate all configure file by lexical order, I guess it is easier to implement. But think about how apache manage multiple web server settings in a folder, and how those settings are independent from each other, that is what we need as a server.