I have a problem when I want to send logs of PFSense (2.5.2 amd64) to EK version 7.14.2

I have a problem when I want to send logs from PFSense (2.5.2 amd64) to EK version 7.14.2
I did configure PFSense to send logs to EK but I did not find the best procedure to configure Elasticsearch and Kibana (7.14.2)

kindly note that my os is ubuntu 20.04

Can you copy your .conf file or debug logs?

Have you configured Filebeat or you are planning to use syslog as input? Your input is set for Filebeat.
Can you add stdout { codec => rubydebug " } and run LS from the command line?

thanks indeed for your reply ,
PFSense send syslog to Logstash,
So I want to use Syslog as input

You can use pure TCP/UDP plugin

input {  
  tcp {
    type => "syslog"
    port => 514
  }

  udp {
    type => "syslog"
    port => 514
  }
}

or Syslog plugin(recommended) which will listen on both TCP and UDP protocols.

input {
 syslog {
   port => 514
 }
}

After that you will receive a message, which will be need to parse by grok or KV plugin. Do not use JSON plugin/coded for this until you receive a raw massage.

1 Like

i did add stdout { codec => rubydebug " } and i run LS like you suggest me

we have already use this plugin in " /etc/logstash/conf.d/01-inputs.conf " but it doesn't work .

LS=Logstash, FB=Filebeat :slight_smile:

Try this code, change ES and file params as you wish.

input {
 syslog {  port => 514 }
}
filter {

}
output {
    # write to ES
    elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "syslog-test_%{+YYYY.MM}"
    }
	
    # write to a file
    file { path => "/var/log/logstash/syslog_%{+YYYY-MM-dd}.txt" }
	
    # debug 
    stdout { codec => rubydebug }

}
1 Like

we did not see logstash send logs to Elasticsearch . also it doesn't create ( var/log/logstash/syslog_ .txt) and doesn't write logs into

Start Logstash as a process not as the service:
/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/your.conf --path.settings /etc/logstash/

Check does your local host firewall open port
service iptables status
or
firewall-cmd --state

Also you can use tcpdump to check does any traffic coming to your host.

1 Like

we can see logs arrive to logstash and written to file (syslog_.txt)
But it can not do the parsing for the raw messages because we did not have grok for pfsense 2.5.2

Can you copy a message? 1-2 lines.

Can you put the message as text not as the picture?

thank you for your response . this is the message as text

<134>1 2022-02-22T10:59:15.721388+01:00 pfSense.home.arpa filterlog 26305 - - 4,,,1000000103,em0,match,block,in,4,0x0,,242,40756,0,DF,6,tcp,40,2.2.2.2,1.1.1.1,44925,443,0,A,,441544382,62500,,

Is only this type of message? This looks like TCP, the message on the picture is NGINX.

There is GitHub - pfelk/pfelk: pfSense/OPNsense + Elastic Stack for pfSense/OPNsense firewall traffic. However, your lines seems a little different types.

thank you for your cooperation.
By trying to decrypt and identify each message field for parsing. I did not find all the fields and it corresponds to what exactly . for example ,

message " <134>1 2022-02-22T10:59:15.721388+01:00 pfSense.home.arpa filterlog 26305 - - 4,,,1000000103,em0,match,block,in,4,0x0,,242,40756,0,DF,6,tcp,40,2.2.2.2,1.1.1.1,44925,443,0,A,,441544382,62500,,"

i find only

have you idea please of the other fields of message .

This is version for pfelk/conf.d/01-inputs.conf
<(?<eventid>.*)>%{INT:sub_rule} (%{SYSLOGTIMESTAMP:[event][created]}|%{TIMESTAMP_ISO8601:[event][created]})\s(%{SYSLOGHOST:[host][name]}\s+)?(%{PROG:[process][name]}\:|%{PROG:[process][name]})\s*?(\[)?(%{POSINT:[process][pid]}(\]:)?)?\s*(\-\s*\-)?\s*%{GREEDYDATA:filter_message}

05-apps.conf is used to parse filter_message

if [process][name] =~ /^filterlog$/ {
 grok {
      patterns_dir => [ "/etc/pfelk/patterns" ]
      match => [ "filter_message", "%{PF_LOG_ENTRY}" ]
    }

PF_LOG_ENTRY you will find in patterns, line 53

github.com/pfelk/pfelk/blob/main/etc/pfelk/patterns/pfelk.grok

I am assuming you will reorganize logic since some fields were changed. Go step by step.

1 Like

thank you very much for your help .
Can you please expalin me with more details because i don't find the best solution for my problem. ( i don't have the same files like PFELK ) .

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.