Hi,
I am trying to parse the logs and sending it to Zabbix.
I am using centOS.
In logstash.yml file, I have configured only 1 setting in pipeline settig section:
path.config: /etc/logstash/conf.d/*.conf
Everything else is default:
[2018-08-30T08:36:52,357][DEBUG][logstash.runner ] pipeline.id: "main"
[2018-08-30T08:36:52,357][DEBUG][logstash.runner ] pipeline.system: false
[2018-08-30T08:36:52,357][DEBUG][logstash.runner ] pipeline.workers: 1
[2018-08-30T08:36:52,357][DEBUG][logstash.runner ] pipeline.output.workers: 1
[2018-08-30T08:36:52,357][DEBUG][logstash.runner ] pipeline.batch.size: 125
[2018-08-30T08:36:52,357][DEBUG][logstash.runner ] pipeline.batch.delay: 5
[2018-08-30T08:36:52,357][DEBUG][logstash.runner ] pipeline.unsafe_shutdown: false
[2018-08-30T08:36:52,358][DEBUG][logstash.runner ] pipeline.reloadable: true
Below is my pipelines.yml file:
- pipeline.id: xx-pipeline
path.config: "/etc/logstash/conf.d/xxx.conf"
pipeline.workers: 2
- pipeline.id: yy-pipeline
path.config: "/etc/logstash/conf.d/yyy.conf"
pipeline.workers: 2
- pipeline.id: zz-pipeline
path.config: "/etc/logstash/zzz.conf"
pipeline.workers: 2
- pipeline.id: aa-pipeline
path.config: "/etc/logstash/conf.d/aaa.conf"
pipeline.workers: 2
first 2 config files are sending data to Elasticsearch and I am able to see logstash data in Kibana.
last 2 config files are sending data to zabbix server and I am not able to see any data in Zabbix server.
I traced logstash logs and below is what I found:
[2018-08-30T08:36:52,392][DEBUG][logstash.config.source.multilocal] Reading pipeline configurations from YAML {:location=>"/etc/logstash/pipelines.yml"}
[2018-08-30T08:36:52,405][WARN ][logstash.config.source.multilocal] Ignoring the 'pipelines.yml' file because modules or command line options are specified
[2018-08-30T08:37:05,894][DEBUG][logstash.pipeline ] filter received {"event"=>{"@version"=>"1", "host"=>"bbb", "path"=>"cccccccc/server.log", "@timestamp"=>2018-08-30T06:37:05.877Z, "message"=>"2018-08-30 00:00:14 CEST [MED.0110.0028E] [MED1250472912-66330] <-- Proxy service: MND_CORPORATE_M4M_YSD_HYBRIS_ORDER_CREATION_WS_VS(1.0). Error in the Mediator outbound client. Web service processing error; more details in the web service error log on provider side (UTC timestamp 20180829220014; Transaction ID 5B87FE590FF91C80E1008000C0A88D71)"}}
[2018-08-30T08:37:09,642][DEBUG][logstash.pipeline ] Pushing flush onto pipeline {:pipeline_id=>"main", :thread=>"#<Thread:0x1e7cf3dc@/usr/share/logstash/logstash-core/lib/logstash/pipeline.rb:290 sleep>"}