PFsense 2.3.3 and ELK

Elastic Stack Logstash
1

After I successfully integrated the logs from my Synology. I now go over to my pfsense. There a some tutorials out in the www. Unfortunately none of them is working properly or just with old versions.

I'm following the tutorial from: http://pfelk.3ilson.com/

At the moment I recieve the logs in the ELK stack but not parsed correctly. The biggest problem is probably the eventid.

filter {  
  if "PFSense" in [tags] {
    grok {
      add_tag => [ "firewall" ]
      match => [ "message", "<(?<evtid>.*)>(?<datetime>(?:Jan(?:uary)?|Feb(?:ruary)?|Mar(?:ch)?|Apr(?:il)?|May|Jun(?:e)?|Jul(?:y)?|Aug(?:ust)?|Sep(?:tember)?|Oct(?:ober)?|Nov(?:ember)?|Dec(?:ember)?)\s+(?:(?:0[1-9])|(?:[12][0-9])|(?:3[01])|[1-9]) (?:2[0123]|[01]?[0-9]):(?:[0-5][0-9]):(?:[0-5][0-9])) (?<prog>.*?): (?<msg>.*)" ]
    }
    mutate {
      gsub => ["datetime","  "," "]
    }
    date {
      match => [ "datetime", "MMM dd HH:mm:ss" ]
      timezone => "America/New_York"
    }
    mutate {
      replace => [ "message", "%{msg}" ]
    }
    mutate {
      remove_field => [ "msg", "datetime" ]
    }
}
if [prog] =~ /^filterlog$/ {  
    mutate {
      remove_field => [ "msg", "datetime" ]
    }
    grok {
      patterns_dir => "/etc/logstash/conf.d/patterns"
      match => [ "message", "%{PFSENSE_LOG_DATA}%{PFSENSE_IP_SPECIFIC_DATA}%{PFSENSE_IP_DATA}%{PFSENSE_PROTOCOL_DATA}",
         "message", "%{PFSENSE_LOG_DATA}%{PFSENSE_IPv4_SPECIFIC_DATA_ECN}%{PFSENSE_IP_DATA}%{PFSENSE_PROTOCOL_DATA}" ]
    }
    mutate {
      lowercase => [ 'proto' ]
    }
    geoip {
      add_tag => [ "GeoIP" ]
      source => "src_ip"
      # Optional GeoIP database




      # Comment out the below if you do not wise to utilize and omit last three steps dealing with (recommended) suffix
      database => "/etc/logstash/GeoLite2-City.mmdb"
    }
  }
}

I made a picture from the view in Kibana at the moment. I tried with: BASE10NUM but that doesn't work.

pfsense1.jpg1455×490 66.6 KB

Can someone help me. This is going to be more difficult I think. But I have all the grok patterns from the tutorial and config. Maybe I just have to make a few adjustments.

2

This is a bit of a messy grok pattern. Why not use the inbuilt time patterns?

3

It looks like you have a kv filter somewhere that creates a field named <134>Apr 27 20. Start by fixing that.

Next time save your screenshots and PNG instead of JPEG to make them more readable.

4

warkolm, inbuilt time patterns, what do you mean with that?

Can you do an example?

@Magnus I have checked it again. I can't see any KV filter in the whole code. Did you look at the website I mentioned? I use the same code as he does.

5

warkolm, inbuilt time patterns, what do you mean with that?

He's talking about the stock grok patterns that ship with Logstash, for example these: https://github.com/logstash-plugins/logstash-patterns-core/blob/master/patterns/grok-patterns

I have checked it again. I can't see any KV filter in the whole code.

What about the rest of your configuration? What's the result of grep kv /etc/logstash/conf.d?

6

What about the rest of your configuration? What's the result of grep kv /etc/logstash/conf.d?
When i do that, i recieve: No file or folder has been found?

I checked if there is any KV filter in the files but I can't find anything. Any other ideas? I will check that today in the evening what warkolm is writing.

7

I found the problem after hours of searching and trying. If there is someone else who needs with Pfsense 2.3+ and CentOS 7 on ELK Stack 5.2+ you can contact me here or go to my website. I'm going to publish there soon a tutorial for CentOS and ELK. www.hs-x.ch

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.