Currently I use logstash to operate pipeline.yml, the contents are as follows, according to my understanding, a line of pipeline.id should be a thread, it should be independent, but recently I found that there is a problem with one of the pipeline.id, such as its output endpoint is offline, then the result is that the entire pipeline.yml will not work properly.
- pipeline.id: abp_auditlogs
path.config: "/etc/logstash/conf.d/abp_auditlogs.conf"
- pipeline.id: lonsid_k8s_dev
path.config: "/etc/logstash/conf.d/lonsid_k8s_dev.conf"
- pipeline.id: lonsid_k8s_prd
path.config: "/etc/logstash/conf.d/lonsid_k8s_prd.conf"
- pipeline.id: ack_k8s_dev
path.config: "/etc/logstash/conf.d/ack_k8s_dev.conf"
- pipeline.id: ack_k8s_prd
path.config: "/etc/logstash/conf.d/ack_k8s_prd.conf"
- pipeline.id: iiot
path.config: "/etc/logstash/conf.d/iiot.conf"
- pipeline.id: scheduler
path.config: "/etc/logstash/conf.d/abp_schedulerlogs.conf"
- pipeline.id: mall_log
path.config: "/etc/logstash/conf.d/mall_log.conf"
- pipeline.id: iot_masa_pre_prd
path.config: "/etc/logstash/conf.d/iot_masa_pre_prd.conf"
According to forum searches, I found that logstash has multiple architectural patterns
The documentation uses config.string, but I don't see any difference from my use of path.config, why is it considered isolated mode? Other patterns are similar, I don't understand.
# config/pipelines.yml
- pipeline.id: intake
config.string: |
input { beats { port => 5044 } }
output { pipeline { send_to => [es, http] } }
- pipeline.id: buffered-es
queue.type: persisted
config.string: |
input { pipeline { address => es } }
output { elasticsearch { } }
- pipeline.id: buffered-http
queue.type: persisted
config.string: |
input { pipeline { address => http } }
output { http { } }
My purpose is to want logstash to keep running stably.
You can use the output isolator pattern to prevent Logstash from becoming blocked if one of multiple outputs experiences a temporary failure.