Is it possible to update a logstash conf file without restarting the logstash instance in the latest 2.1 version? e.g. if I am reading from one kafka topic, and in the meantime another topic has pushed some data, I need to start listening to both the topics. I can add the second one to white_list, but for the change to take effect I have to restart ls. Also, I don't want to spawn new instance for each topic.
Is it possible to update a logstash conf file without restarting the logstash instance in the latest 2.1 version?
Unfortunately not.
Thank you. I need one more suggestion since it is not possible to update the conf file on the fly. If we have n number of input topics and each one has same load, is it better to handle all of those using only one LS instance using conditional statements or to spawn a different conf file for each topic?
Also is there any upper limit on the number of input kafka topics to listen to using only one LS instance, or does it totally depend upon the nature of the stream?
Thank you. I need one more suggestion since it is not possible to update the conf file on the fly. If we have n number of input topics and each one has same load, is it better to handle all of those using only one LS instance using conditional statements or to spawn a different conf file for each topic?
That depends on the value of n, I'd say. A Logstash instance has a single event pipeline so a clogged output will affect all messages, plus there are probably per-instance bottlenecks that one would be affected by even if the inputs and filters are threaded. That speaks for using multiple instances, but for n > 5 you're probably going to have a too large overhead.
Also is there any upper limit on the number of input kafka topics to listen to using only one LS instance, or does it totally depend upon the nature of the stream?
I expect the number is only limited by memory and other resources. For example, I suspect that each kafka input will use at least one thread and one file descriptor.