[LogStash::Runner] agent - failed to fetch pipeline configuration


(yurizal) #1

Hi
I followed the tutorial on "https://extelligenceblog.it/2017/07/23/elastic-stack-suricata-idps-and-pfsense-firewall-part-3-logstash-pipeline-additions-suricata-alerts/"

this step is affected error (/usr/share/logstash/bin/logstash -f ./10-pfsense-filter.conf)
what causes and resolves the error

[ERROR] 2018-02-21 05:47:43.380 [LogStash::Runner] agent - failed to fetch pipeline configuration {:message=>"No config files found: ./10-pfsense-filter.conf. Can you make sure this path is a logstash config file?"}

ELK Stack 5.6.7, Filebeat 6.2

[Logstash]

cd /etc/logstash/conf.d
#tcp syslogs stream via 5140
input {
 tcp {
 type => "syslog"
 port => 5140
 }
}
#udp syslogs stream via 5140
input {
 udp {
 type => "syslog"
 port => 5140
 }
}

input {
 beats {
 port => 5044
 }
}

sudo nano 10-pfsense-filter.conf

#input { stdin { } }
filter {
   if [type] =="suricataIDPS" {
      json {
         source => "message"
      }
      date {
         match => [ "timestamp", "ISO8601" ]
      }
   }
}
#output { stdout { codec => rubydebug } }

Filebeat

filebeat.prospectors: 
  - 
    fields: 
      tags: 
        - SuricataIDPS
        - JSON
      document_type: suricataIDPS
    fields_under_root: true
    type: log
    paths: 
      - /var/log/suricata/*/eve.json*

output.logstash:
  hosts: ["x.x.x.x:5044"]

logging.files: 
  keepfiles: 7
  name: filebeat.log
  path: /var/log/filebeat
logging.to_files: true

(Andrew Kroh) #2

The tutorial has you write the file to /etc/logstash/conf.d/10-pfsense-filter.conf. What directory are you running the command from? You must be in /etc/logstash/conf.d.

If you use the absolute path to the config file then you can run the Logstash command from directory. For example:

/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/10-pfsense-filter.conf


(yurizal) #3

thank you sir, i also tried it and it works, but it is empty. finally I try
/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/10-pfsense-filter.conf

[ERROR] 2018-02-24 11:13:55.726 [[main]<tcp] pipeline - A plugin had an unrecoverable error. Will restart this plug Plugin: <LogStash::Inputs::Tcp type=>"syslog", port=>5140, id=>"de54bfa1d7f7a81e92033d2704be9e04a422f98b-1", enabecs::Line id=>"line_b840a9bd-02ee-46d6-90a3-ca66dc5be4cd", enable_metric=>true, charset=>"UTF-8", delimiter=>"\n">,de=>"server", proxy_protocol=>false, ssl_enable=>false, ssl_verify=>true, ssl_key_passphrase=><password>> Error: Address already in use

I look at the log:
tail -f logstash-plain.log success


(system) #4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.