I have a problem when I want to send logs of PFSense (2.5.2 amd64) to EK version 7.14.2

Elastic Stack Logstash
1

I have a problem when I want to send logs from PFSense (2.5.2 amd64) to EK version 7.14.2
I did configure PFSense to send logs to EK but I did not find the best procedure to configure Elasticsearch and Kibana (7.14.2)

2

kindly note that my os is ubuntu 20.04

3

Can you copy your .conf file or debug logs?

5

Have you configured Filebeat or you are planning to use syslog as input? Your input is set for Filebeat.
Can you add stdout { codec => rubydebug " } and run LS from the command line?

6

thanks indeed for your reply ,
PFSense send syslog to Logstash,
So I want to use Syslog as input

1 Like
7

You can use pure TCP/UDP plugin

input {  
  tcp {
    type => "syslog"
    port => 514
  }

  udp {
    type => "syslog"
    port => 514
  }
}

or Syslog plugin(recommended) which will listen on both TCP and UDP protocols.

input {
 syslog {
   port => 514
 }
}

After that you will receive a message, which will be need to parse by grok or KV plugin. Do not use JSON plugin/coded for this until you receive a raw massage.

1 Like
8

i did add stdout { codec => rubydebug " } and i run LS like you suggest me

image
image920×301 143 KB

we have already use this plugin in " /etc/logstash/conf.d/01-inputs.conf " but it doesn't work .

9

LS=Logstash, FB=Filebeat :slight_smile:

Try this code, change ES and file params as you wish.

input {
 syslog {  port => 514 }
}
filter {

}
output {
    # write to ES
    elasticsearch {
    hosts => ["http://localhost:9200"]
    index => "syslog-test_%{+YYYY.MM}"
    }
	
    # write to a file
    file { path => "/var/log/logstash/syslog_%{+YYYY-MM-dd}.txt" }
	
    # debug 
    stdout { codec => rubydebug }

}
1 Like
10

we did not see logstash send logs to Elasticsearch . also it doesn't create ( var/log/logstash/syslog_ .txt) and doesn't write logs into

11

Start Logstash as a process not as the service:
/usr/share/logstash/bin/logstash -f /etc/logstash/conf.d/your.conf --path.settings /etc/logstash/

Check does your local host firewall open port
service iptables status
or
firewall-cmd --state

Also you can use tcpdump to check does any traffic coming to your host.

1 Like
14

we can see logs arrive to logstash and written to file (syslog_.txt)
But it can not do the parsing for the raw messages because we did not have grok for pfsense 2.5.2

15

Can you copy a message? 1-2 lines.

16

MicrosoftTeams-image (1)
MicrosoftTeams-image (1)1023×448 107 KB

17

Can you put the message as text not as the picture?

18

thank you for your response . this is the message as text

<134>1 2022-02-22T10:59:15.721388+01:00 pfSense.home.arpa filterlog 26305 - - 4,,,1000000103,em0,match,block,in,4,0x0,,242,40756,0,DF,6,tcp,40,2.2.2.2,1.1.1.1,44925,443,0,A,,441544382,62500,,

19

Is only this type of message? This looks like TCP, the message on the picture is NGINX.

There is GitHub - pfelk/pfelk: pfSense/OPNsense + Elastic Stack for pfSense/OPNsense firewall traffic. However, your lines seems a little different types.

20

thank you for your cooperation.
By trying to decrypt and identify each message field for parsing. I did not find all the fields and it corresponds to what exactly . for example ,

message " <134>1 2022-02-22T10:59:15.721388+01:00 pfSense.home.arpa filterlog 26305 - - 4,,,1000000103,em0,match,block,in,4,0x0,,242,40756,0,DF,6,tcp,40,2.2.2.2,1.1.1.1,44925,443,0,A,,441544382,62500,,"

i find only

CAPTURE
CAPTURE746×504 89.5 KB

have you idea please of the other fields of message .

21

This is version for pfelk/conf.d/01-inputs.conf
<(?<eventid>.*)>%{INT:sub_rule} (%{SYSLOGTIMESTAMP:[event][created]}|%{TIMESTAMP_ISO8601:[event][created]})\s(%{SYSLOGHOST:[host][name]}\s+)?(%{PROG:[process][name]}\:|%{PROG:[process][name]})\s*?(\[)?(%{POSINT:[process][pid]}(\]:)?)?\s*(\-\s*\-)?\s*%{GREEDYDATA:filter_message}

05-apps.conf is used to parse filter_message

if [process][name] =~ /^filterlog$/ {
 grok {
      patterns_dir => [ "/etc/pfelk/patterns" ]
      match => [ "filter_message", "%{PF_LOG_ENTRY}" ]
    }

PF_LOG_ENTRY you will find in patterns, line 53

github.com/pfelk/pfelk/blob/main/etc/pfelk/patterns/pfelk.grok

I am assuming you will reorganize logic since some fields were changed. Go step by step.

1 Like
22

thank you very much for your help .
Can you please expalin me with more details because i don't find the best solution for my problem. ( i don't have the same files like PFELK ) .

23

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.