Filebeat-panw Module timezone issues

Elastic Stack Beats
1

Hello, I am sorry to re-hash the same issue others have had but I can't seem to get the fixes they have done to work for my environment and it has driven me crazy trying to figure it out.

So right now, when I setup FileBeat panw module and send syslog data from our PaloAlto to the filebeat module the time is always 4 hours prior to what real time is. From the document to setup I need to remove this functionality but I can't seem to get that right. Here is what I have:

filebeat.overwirte_pipelines: true

#filebeat.config.modules:
  # Glob pattern for configuration loading
  #path: ${path.config}/modules.d/*.yml

  # Set to true to enable config reloading
  #reload.enabled: false

  # Period on which files under path should be checked for changes
  #reload.period: 10s

filebeat.modules:
  - module: panw
    panos:
      enabled: true
      var.syslog_host: 0.0.0.0
      var.syslog_port: 9001
      #var.convert_timezone: true
processors:
#  - drop_fields.fields: ["event.timezone"]
#  - add_fields:
#    target: event
#    fields:
#      timezone: 'America/New_York'
  - drop_fields:
    fields: ["event.timezone"]
    ignore_missing: true

As you can see from the config I have tried a lot of items and none of them seem to work. After every change I make sure I delete the pipelines and data out so that I am working with only fresh data. I have also set timezone on my server to be UTC, +4 from UTC and nothing changes the data. It is always 4 hours in the past when looking at it.

I get this same data a different method using Logstash and I set the timezone to America/New_York when the data is first ingested. When looking at the index all data is the right time. I just don't know how to do the same thing with Filebeat that I am doing with Logstash.

Any help on my config or leading me into a different place to make this work would be great.

This is Filebeat 7.4.0

2

Here is the JSON from the even while looking at:

 {
      "_index": "filebeat-7.4.0-2019.10.03-000001",
      "_type": "_doc",
      "_id": "9ShFk20BIX4xqh5eB-MC",
      "_version": 1,
      "_score": null,
      "_source": {
        "server": {
          "port": 80,
          "ip": "X.X.X.21"
        },
        "agent": {
          "hostname": "cvelk04",
          "id": "b4086736-2908-4b26-a0d5-1a680cf26b36",
          "type": "filebeat",
          "ephemeral_id": "bc427f71-7208-42fb-9ed9-d62b2f259131",
          "version": "7.4.0"
        },
        "log": {
          "original": "1,2019/10/03 16:17:17,001701015958,THREAT,url,1,2019/10/03 16:17:17,X.X.X.31,X.X.X.21,X.X.X.30,X.X.X.21,Level 4 - Social Network,cvi\\XXXX,,ssl,vsys1,Trust-LAN,XXXX
    ,ethernet1/6,ethernet1/8,XXXX SOD Syslog Forwarding,2019/10/03 16:17:17,217509,1,56425,80,38338,80,0x50f000,tcp,alert,\"academy.usa.com/\",(9999),business-and-economy,informational,client-to-server,487383760,0x0,X.X.0.0-X.X.255.255,X.X.0.0-X.X.255.255,0,,0,,,0,,,,,,,,0,0,0,0,0,vsys1,Palo,,,,connect,0,,0,,N/A,unknown,AppThreat-0-0,0x0",
          "level": "informational",
          "source": {
            "address": "X.X.X.30:42511"
          }
        },
        "destination": {
          "geo": {
            "country_iso_code": "X.X.0.0-X.X.255.255"
          },
          "nat": {
            "port": 80,
            "ip": "X.X.X.21"
          },
          "address": "X.X.X.21",
          "port": 80,
          "ip": "X.X.X.21"
        },
        "syslog": {
          "priority": 14,
          "facility": 1,
          "severity_label": "Informational",
          "facility_label": "user-level"
        },
        "source": {
          "geo": {
            "country_iso_code": "X.X.0.0-X.X.255.255"
          },
          "nat": {
            "port": 38338,
            "ip": "X.X.X.30"
          },
          "address": "X.X.X.31",
          "port": 56425,
          "ip": "X.X.X.31",
          "user": {
            "name": "cvi\\XXXXXX"
          }
        },
        "error": {
          "message": [
            "field [timestamp] not present as part of path [timestamp]"
          ]
        },
        "panw": {
          "panos": {
            "flow_id": "217509",
            "ruleset": "Level 4 - Social Network",
            "destination": {
              "nat": {
                "port": 80,
                "ip": "X.X.X.21"
              },
              "zone": "CUSA",
              "interface": "ethernet1/8"
            },
            "threat": {
              "resource": "academy.usa..com/",
              "name": "URL-filtering",
              "id": "9999"
            },
            "source": {
              "nat": {
                "port": 38338,
                "ip": "X.X.X.30"
              },
              "zone": "Trust-LAN",
              "interface": "ethernet1/6"
            },
            "url": {
              "category": "business-and-economy"
            },
            "network": {
              "nat": {
                "community_id": "1:bb8ofHDzxfmHmh/RHfamWVAcNLM="
              }
            }
          }
        },
        "network": {
          "community_id": [
            "1:XH8MzKHVYgmIkgvS36LxmxP10K8=",
            "1:bb8ofHDzxfmHmh/RHfamWVAcNLM="
          ],
          "application": "ssl",
          "transport": "tcp",
          "direction": "inbound"
        },
        "observer": {
          "hostname": "Palo",
          "serial_number": "001701015958"
        },
        "hostname": "Palo..com",
        "ecs": {
          "version": "1.1.0"
        },
        "related": {
          "ip": [
            "X.X.X.31",
            "X.X.X.21",
            "X.X.X.30",
            "X.X.X.21"
          ]
        },
        "host": {
          "hostname": "cvelk04",
          "os": {
            "kernel": "3.10.0-957.10.1.el7.x86_64",
            "codename": "Core",
            "name": "CentOS Linux",
            "family": "redhat",
            "version": "7 (Core)",
            "platform": "centos"
          },
          "containerized": false,
          "name": "cvelk04",
          "id": "83d7ca5856ce40d3b90a383e288ae932",
          "architecture": "x86_64"
        },
        "client": {
          "port": 56425,
          "ip": "X.X.X.31",
          "user": {
            "name": "cvi\\XXXXXX"
          }
        },
        "event": {
          "severity": 5,
          "timezone": "+04:00",
          "created": "2019-10-03T20:17:17.000+04:00",
          "module": "panw",
          "action": "url_filtering",
          "category": "security_threat",
          "dataset": "panw.panos",
          "outcome": "alert"
        },
        "fileset": {
          "name": "panos"
        },
        "url": {
          "original": "academy.usa..com/"
        },
        "tags": [
          "pan-os"
        ],
        "labels": {
          "container_page": true,
          "nat_translated": true,
          "temporary_match": true
        },
        "input": {
          "type": "syslog"
        },
        "@timestamp": "2019-10-03T16:17:17.000Z",
        "service": {
          "type": "panw"
        }
      },
      "fields": {
        "suricata.eve.timestamp": [
          "2019-10-03T16:17:17.000Z"
        ],
        "@timestamp": [
          "2019-10-03T16:17:17.000Z"
        ],
        "event.created": [
          "2019-10-03T16:17:17.000Z"
        ]
      },
      "sort": [
        1570119437000
      ]
    }
3

Hi,

Your log line works for me:

$ cat u.log
1,2019/10/03 16:17:17,001701015958,THREAT,url,1,2019/10/03 16:17:17,10.255.255.31,10.255.255.21,10.255.255.30,10.255.255.21,Level 4 - Social Network,cvi\\XXXX,,ssl,vsys1,Trust-LAN,XXXX,ethernet1/6,ethernet1/8,XXXX SOD Syslog Forwarding,2019/10/03 16:17:17,217509,1,56425,80,38338,80,0x50f000,tcp,alert,\"academy.usa.com/\",(9999),business-and-economy,informational,client-to-server,487383760,0x0,X.X.0.0-X.X.255.255,X.X.0.0-X.X.255.255,0,,0,,,0,,,,,,,,0,0,0,0,0,vsys1,Palo,,,,connect,0,,0,,N/A,unknown,AppThreat-0-0,0x0

with this change to filebeat.yml:

diff --git a/x-pack/filebeat/filebeat.yml b/x-pack/filebeat/filebeat.yml
index d02b4d161..968b2cee2 100644
--- a/x-pack/filebeat/filebeat.yml
+++ b/x-pack/filebeat/filebeat.yml
@@ -178,6 +178,9 @@ processors:
   - add_cloud_metadata: ~
   - add_docker_metadata: ~
   - add_kubernetes_metadata: ~
+  - drop_fields:
+        fields: ["event.timezone"]
+        ignore_missing: true

I get this document:

{
    [...]
    "log": {
      "original": "1,2019/10/03 16:17:17,001701015958,THREAT,url,1,2019/10/03 16:17:17,10.255.255.31,10.255.255.21,10.255.255.30,10.255.255.21,Level 4 - Social Network,cvi\\\\XXXX,,ssl,vsys1,Trust-LAN,XXXX,ethernet1/6,ethernet1/8,XXXX SOD Syslog Forwarding,2019/10/03 16:17:17,217509,1,56425,80,38338,80,0x50f000,tcp,alert,\\\"academy.usa.com/\\\",(9999),business-and-economy,informational,client-to-server,487383760,0x0,X.X.0.0-X.X.255.255,X.X.0.0-X.X.255.255,0,,0,,,0,,,,,,,,0,0,0,0,0,vsys1,Palo,,,,connect,0,,0,,N/A,unknown,AppThreat-0-0,0x0\n",
      "level": "informational"
    },
    [...]
  "fields": {
    "@timestamp": [
      "2019-10-03T16:17:17.000Z"
    ],
    "event.created": [
      "2019-10-03T16:17:17.000Z"
    ]
  }
}

In your config the drop_fields processor is not properly indented, and you can tell its not working because the event.timezone field is still present in the resulting document.

Can you try again making sure that the processor is indented correctly?

Also take into account that by default Kibana will convert timestamps to the timezone reported by your browser's. You can change that under Advanced Settings (dateFormat:tz). I also recommend that you add an Z at the end of the dateFormat field so that it prints the timezone offset.

4

Thanks for the help, I took a look at the config and matched what you posted.

#================================ Processors =====================================

# Configure processors to enhance or manipulate events generated by the beat.

processors:
  - add_host_metadata: ~
  - add_cloud_metadata: ~
  - drop_fields:
        fields: ["evemt.timezone"]
        ignore_missing: true

I can see that event.timezone is being removed but the time is still showing as 4 hours ago.

So I made the change to kibana to be UTC time instead of browser and that does fix time for the filebeat data but it breaks all of the other data that is being indexed with Elastic. My flow data using logstash was 4 hours in the future instead of current. Is there any way I can fix it just for this filebeat data?

5

You can add a condition:

processors:
  - drop_fields:
      fields: [ "event.timezone" ]
      ignore_missing: true
      when.equals:
        event.dataset: "panw.panos"

Make sure you got the fields: right, in your message it reads evemts

1 Like
6

Sorry I explained that bad. My issue isn't the filebeats removing the event.timezone. This filebeats only exists to ingest that panos data for the SIEM app.

My concern is that even with removing that field the data still was converted to 4 hours in the past. After changing the Kibana advance setting to UTC it set the data to the right time, but that also broke all the other data sources that come into ELK from various other applications. Their time instead of being correct was set to 4 hours in the future, or UTC time.

7

You're right, that won't fix it with the panw module. There's a bug on how the dates are handled.

We have a fix ready:

You can replace the module/panw/panos/ingest/pipeline.yml with the fixed one from this PR, which does the right thing with dates.

1 Like
8

Adrian, thank you for this! This did fix my issue. in order to get it to work I had to delete everything and re-install filebeat. Once that was done everything came right back up and the data is coming in with the correct time.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.