# Filebeat 7.7 panw module sends THREAT, but not TRAFFIC logs
**URL:** https://discuss.elastic.co/t/filebeat-7-7-panw-module-sends-threat-but-not-traffic-logs/239792
**Category:** Beats
**Tags:** filebeat
**Created:** [July 3, 2020, 11:46am UTC](https://discuss.elastic.co/t/filebeat-7-7-panw-module-sends-threat-but-not-traffic-logs/239792 "2020-07-03T11:46:13Z")
**Posts on this page:** 9
**Page:** 1
### Author:  [@nverrill](https://discuss.elastic.co/u/nverrill)
#### Post date: [July 3, 2020, 11:46am UTC](https://discuss.elastic.co/t/filebeat-7-7-panw-module-sends-threat-but-not-traffic-logs/239792/1 "2020-07-03T11:46:13Z")
I am testing Filebeat 7.7 with the panw module, and am receiving THREAT type logs, but not TRAFFIC type logs. Elasticsearch version is also 7.7.
filebeat.yml config:
```auto
filebeat.config.modules:
path: ${path.config}/modules.d/*.yml
output.elasticsearch:
hosts: ["host1", "host2"]
loadbalance: true
username: "user"
password: "pass"
protocol: https
ssl.certificate_authorities: ["/path/ca.pem"]
ssl.key: "/path/ssl.key"
ssl.key_passphrase: 'passphrase'
ssl.certificate: "/path/ssl.crt"
```
panw.yml config:
```auto
- module: panw
panos:
enabled: true
var:
input: "file"
paths: ["/path/to/file.log"]
```
I'm using pretty basic config here without customization, the log file contains both threat and traffic logs. Threat logs ingest without issue, traffic logs never show up.
Any ideas where to look for logs that may point to the problem? filebeat and elasticsearch logs don't show errors that I can find.
---
### Author:  [@kvch](https://discuss.elastic.co/u/kvch)
#### Post date: [July 6, 2020, 3:27pm UTC](https://discuss.elastic.co/t/filebeat-7-7-panw-module-sends-threat-but-not-traffic-logs/239792/2 "2020-07-06T15:27:55Z")
Could you please share the debug logs Filebeat (`./filebeat -e -d "*"`)?
---
### Author:  [@nverrill](https://discuss.elastic.co/u/nverrill)
#### Post date: [July 13, 2020, 12:17pm UTC](https://discuss.elastic.co/t/filebeat-7-7-panw-module-sends-threat-but-not-traffic-logs/239792/3 "2020-07-13T12:17:08Z")
@kvch Using the debug parameter you showed me, I was able to see immediately that it is being parsed incorrectly. There is an IP in the "message\_subtype" field, which of course gets dropped as only the TRAFFIC or THREAT message\_subtype's are processed in this module.
Here's a sample of the stdout with debug parameter:
```auto
Publish event: {
"@timestamp": "2020-07-06T18:52:34.990Z",
"@metadata": {
"beat": "filebeat",
"type": "_doc",
"version": "7.7.0",
"pipeline": "filebeat-7.7.0-panw-panos-pipeline"
},
"observer": {
"serial_number": "TRAFFIC"
},
"log": {
"offset": 102246,
"file": {
"path": "/path/to/file.log"
}
},
"tags": [
"pa_test",
"pan-os"
],
"input": {
"type": "log"
},
"host": {
"name": "hostname.local.com"
},
"_temp_": {
"message_subtype": "192.168.1.1",
"generated_time": "0.0.0.0",
"message_type": "drop"
},
"ecs": {
"version": "1.5.0"
},
"agent": {
"ephemeral_id": "8e98e9ea-f1e0-477d-a4f0-979ded422c11",
"hostname": "hostname.local.com",
"id": "240caa00-2e60-42e3-bf40-5b99ac213836",
"version": "7.7.0",
"type": "filebeat"
},
"message": "Jul 1 05:00:00 firewall 2020/07/01 05:00:00,0008C100362,TRAFFIC,drop,192.168.1.1,192.168.1.2,0.0.0.0,0.0.0.0,RuleName,,,not-applicable,vsys1,Zone1,Zone2,ethernet1/11,,sharedPanorama,0,1,59589,443,0,0,0x4000,tcp,deny,66,1,2020/07/01 05:00:00,0,any,66,0",
"event": {
"module": "panw",
"dataset": "panw.panos",
"timezone": "-04:00",
"created": "0008C100362"
},
"fileset": {
"name": "panos"
},
"service": {
"type": "panw"
}
}
```
Using this command:
```auto
GET /_ingest/pipeline/filebeat-7.7.0-panw-panos-pipeline
```
I can look at the pipeline, but I'm no expert here. It could be in the pipeline or the painless scripts being called. Wherever the field parsing happens.
Offhand guess, I'd say maybe it's the version of Palo-Alto logs? Or maybe we customized our log outputs so the number or order of fields is different than expected?
---
### Author:  [@kvch](https://discuss.elastic.co/u/kvch)
#### Post date: [July 21, 2020, 8:04am UTC](https://discuss.elastic.co/t/filebeat-7-7-panw-module-sends-threat-but-not-traffic-logs/239792/4 "2020-07-21T08:04:18Z")
Filebeat modules only supported the default log format of these products. What version are you running?
I am looking into the problem to see if we can adjust the module to fit your case.
---
### Author:  [@adrisr](https://discuss.elastic.co/u/adrisr)
#### Post date: [July 21, 2020, 8:39am UTC](https://discuss.elastic.co/t/filebeat-7-7-panw-module-sends-threat-but-not-traffic-logs/239792/5 "2020-07-21T08:39:15Z")
This message is formatted differently than the pipeline expects:
```auto
Jul 1 05:00:00 firewall 2020/07/01 05:00:00,0008C100362,TRAFFIC,drop,192.168.1.1,192.168.1.2,0.0.0.0,0.0.0.0,RuleName,,,not-applicable,vsys1,Zone1,Zone2,ethernet1/11,,sharedPanorama,0,1,59589,443,0,0,0x4000,tcp,deny,66,1,2020/07/01 05:00:00,0,any,66,0",
```
First it needs an extra field at the start, which in our samples is "1,", but even if that's fixed, other fields don't match what the parser expects, here's an example of a valid log:
```auto
Nov 30 16:09:08 PA-220 1,2018/11/30 16:09:07,012801096514,TRAFFIC,end,2049,2018/11/30 16:09:07,192.168.15.207,184.51.253.152,192.168.1.63,184. 51.253.152,new_outbound_from_trust,,,apple-maps,vsys1,trust,untrust,ethernet1/2,ethernet1/1,send_to_mac,2018/11/30 16:09:07,22751,1,55113,443, 16418,443,0x400053,tcp,allow,7734,1758,5976,36,2018/11/30 15:59:04,586,computer-and-internet-info,0,32091112,0x0,192.168.0.0-192.168.255.255,U nited States,0,16,20,tcp-fin,0,0,0,0,,PA-220,from-policy,,,0,,0,,N/A,0,0,0,0
```
You'll see the initial "1" field after the syslog header, and then that there are two timestamp fields while the provided logs have only the first one.
This is the PAN-OS documentation we used to develop the module:
[https://docs.paloaltonetworks.com/pan-os/9-1/pan-os-admin/monitoring/use-syslog-for-monitoring/syslog-field-descriptions/traffic-log-fields.html](https://docs.paloaltonetworks.com/pan-os/9-1/pan-os-admin/monitoring/use-syslog-for-monitoring/syslog-field-descriptions/traffic-log-fields.html)
We support all the versions documented in the above document, but none conforms to the provided log.
Which device / version are you using? Maybe there is some configuration option in the device to select different output formats?
---
### Author:  [@adrisr](https://discuss.elastic.co/u/adrisr)
#### Post date: [July 21, 2020, 8:48am UTC](https://discuss.elastic.co/t/filebeat-7-7-panw-module-sends-threat-but-not-traffic-logs/239792/6 "2020-07-21T08:48:13Z")
Alternatively, if no option is found to adapt the logs to the documentation, you can try configuring CEF output as described in:
[https://docs.paloaltonetworks.com/pan-os/9-1/pan-os-admin/monitoring/use-syslog-for-monitoring/syslog-field-descriptions/custom-logevent-format.html#id1dba4621-ff18-49f6-b613-50f678ce8351](https://docs.paloaltonetworks.com/pan-os/9-1/pan-os-admin/monitoring/use-syslog-for-monitoring/syslog-field-descriptions/custom-logevent-format.html#id1dba4621-ff18-49f6-b613-50f678ce8351)
and use Filebeat's `cef` module to ingest them.
---
### Author:  [@nverrill](https://discuss.elastic.co/u/nverrill)
#### Post date: [July 22, 2020, 4:32pm UTC](https://discuss.elastic.co/t/filebeat-7-7-panw-module-sends-threat-but-not-traffic-logs/239792/7 "2020-07-22T16:32:08Z")
Versions here range from 7.1 to 8.1 which is within compatible ranges according to filebeat docs.
I just confirmed that we customized our traffic output logs. Some fields had been trimmed from the output.
I've now tested a default output and they are ingesting but with the wrong time zone in kibana, which appears to be a known issue where there is no timezone defined in the log itself:
>
>
> Similar to #13308, part of #13877, fixes elastic/beats#13867
> [@Filebeat-panw Module timezone issues](https://discuss.elastic.co/t/filebeat-panw-module-timezone-issues/202193/6):
>
> Sorry I explained that bad. My issue isn't the filebeats removing the event.timezone. This filebeats only exists to ingest that panos data for the SIEM app. My concern is that even with removing that field the data still was converted to 4 hours in the past. After changing the Kibana advance setting to UTC it set the data to the right time, but that also broke all the other data sources that come into ELK from various other applications. Their time instead of being correct was set to 4 hours i…
I'm a little uncertain how to fix this. I tried some of the fixes such as replacing the module/panw/panos/ingest/pipeline.yml in the pull request, as well as upgrading from filebeat 7.7.0 to 7.7.1 and it's still not the correct timezone in Kibana.
Is there something I can provide to help diagnose the issue?
---
### Author:  [@nverrill](https://discuss.elastic.co/u/nverrill)
#### Post date: [July 24, 2020, 6:13pm UTC](https://discuss.elastic.co/t/filebeat-7-7-panw-module-sends-threat-but-not-traffic-logs/239792/8 "2020-07-24T18:13:17Z")
I was able to fix the timezone by using the drop\_fields processor as recommended here:
[https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-module-panw.html#\_time\_zone\_support\_11](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-module-panw.html#_time_zone_support_11)
I thought this was referring to the filebeat panw ingest pipeline processors at first, but the drop\_fields processor config goes in the filebeat.yml. My config ended up looking like this:
```auto
processors:
- drop_fields:
fields: ["event.timezone"]
ignore_missing: true
```
Thanks for the help folks.
---
### Author:  [@system](https://discuss.elastic.co/u/system)
#### Post date: [August 21, 2020, 8:13pm UTC](https://discuss.elastic.co/t/filebeat-7-7-panw-module-sends-threat-but-not-traffic-logs/239792/9 "2020-08-21T20:13:18Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.