Hi All,
I am curious to know running multiple pipelines in logstash
I uncomment the path.config in logstash.yml and keep some pipeline configs in pipelines.yml, how can i add more configs without stopping existing processes and how do I stop a particular pipeline.
First configure Logstash to reload its configuration when it changes with these settings in logstash.yml
config.reload.automatic: true
config.reload.interval: 10s
Whenever you add new pipelines to pipelines.yml they will be started automatically. When you delete (or comment out) a pipeline it will be stopped.
Since you are worried about adding/removing pipelines without restarting Logstash you might also want to consider designing your ingest method to use a queue between Logstash roles such as...
LS Collector --> Redis or Kafka --> LS Processor --> Elasticsearch
The Logstash collector is very simple and almost never needs to be restarted. This gives you the flexibility to modify and restart the Logstash processor instance(s) with no loss of data. Any new data will be queued and handled once the processor is restarted.
Is there anyway i can have specific log for a particular pipeline
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.