Logstash always queue in multiple pipeline

Elastic Stack Elasticsearch
, ,
1

Hi Master,

i'm using logstash 8.8

i'm on stuck in this condtion.

after i created multiple pipeline like this

- pipeline.id: beats
  path.config: "/etc/logstash/conf.d/beats.conf"
  pipeline.workers: 3
- pipeline.id: fortinet
  path.config: "/etc/logstash/conf.d/forti.conf"
  queue.type: persisted

beats running normal, but forti log queueing.
i think they will running parallel. but i'm wrong.

and i reconfigured pipeline.yml to

/etc/logstash/*.conf

forti still queueing.
this log running logstash

[logstash.inputs.udp      ][fw-forti][39698f315ccd0485bdd466944a7eda39915e718be33bf6715e78ed7e43177c25] Starting UDP listener {:address=>"0.0.0.0:5514"}
[logstash.inputs.udp      ][fw-forti][39698f315ccd0485bdd466944a7eda39915e718be33bf6715e78ed7e43177c25] UDP listener started {:address=>"0.0.0.0:5514", :receive_buffer_bytes=>"106496", :queue_size=>"2000"}

forti log not showing in Kibana. even i running in "bin -f"

please help me

2

These are different things, when using pipelines.yml you will have both pipelines running.

Logstash will always have a queue, but this queue can be a memory queue, the default, or a persistent disk queue, as you configured it for the fortinet pipeline.

The queue is not an issue as it will always exist, if you are not receiving any logs for the fortinet pipeline you need to troubleshoot it.

First do you have any log error in Logstash? How does the /etc/logstash/conf.d/forti.conf looks like?

Also, use tcpdump on the server to check if you are indeed receiving logs from your network device, this is the first thing you need to check.

3 
input {
  udp {
    port => 5514
    type => "forti_log"
    codec => plain {
    charset => "ISO-8859-1"
}
}
}

filter  {
 if [type] == "forti_log" {
 kv { 
  source => "message"    
  field_split_pattern => " (?=[A-Za-z0-9]+=)"
}
}


output {
 if [type] == "forti_log" {
   elasticsearch {
   hosts => ["https://elastic:9200"]
   index => "network-forti-%{+YYYY.MM.dd}"
   user => "elastic"
   password => "password"
   ssl => true
   cacert => "/etc/logstash/certs/CA.cer"
   ssl_certificate_verification => false
}
}
}

the reason i using multiple pipeline cause, some log from beats jump in forti indices.

so what i need to do now ?

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.