# Parsing issues with Logstash CSV plugin

**URL:** <https://discuss.elastic.co/t/parsing-issues-with-logstash-csv-plugin/249856>\
**Category:** Logstash\
**Created:** [September 24, 2020, 4:45pm UTC](https://discuss.elastic.co/t/parsing-issues-with-logstash-csv-plugin/249856 "2020-09-24T16:45:45Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![petehalatsis](https://avatars.discourse-cdn.com/v4/letter/p/bbce88/32.png) [@petehalatsis](https://discuss.elastic.co/u/petehalatsis)\
**Post date:** [September 24, 2020, 4:45pm UTC](https://discuss.elastic.co/t/parsing-issues-with-logstash-csv-plugin/249856/1 "2020-09-24T16:45:45Z")

</div>

I have been beating my head on this for a while. My syslog is hitting my if statement and using the columns I have below but they are off by one. I tried adding/removing the first column to account for the shift but it doubles my error somehow. Is there something silly that I am doing wrong here?

(Column reference: [https://docs.paloaltonetworks.com/pan-os/9-0/pan-os-admin/monitoring/use-syslog-for-monitoring/syslog-field-descriptions/threat-log-fields.html](https://docs.paloaltonetworks.com/pan-os/9-0/pan-os-admin/monitoring/use-syslog-for-monitoring/syslog-field-descriptions/threat-log-fields.html))

```auto
if ([message] =~ /THREAT/) {
            csv {
                source => "message"
                columns => [ 
                    "FUTUREUSE", "ReceiveTime", "SerialNumber", "Type", "Threat/ContentType", "FUTUREUSE",
                    "GeneratedTime", "SourceAddress", "DestinationIP", "NATSourceIP", "NATDestinationIP",
                    "RuleName", "SourceUser", "DestinationUser", "Application", "VirtualSystem", "SourceZone",
                    "DestinationZone", "InboundInterface", "OutboundInterface", "LogAction", "FUTUREUSE",
                    "SessionID", "RepeatCount", "SourcePort", "DestinationPort", "NATSourcePort",
                    "NATDestinationPort", "Flags", "Protocol", "Action", "URL/Filename", "ThreatID",
                    "Category", "Severity", "Direction", "SequenceNumber", "ActionFlags", "SourceLocation",
                    "DestinationLocation", "FUTUREUSE", "ContentType", "PCAP_ID", "FileDigest", "Cloud",
                    "URLIndex", "User Agent", "FileType", "X-Forwarded-For", "Referer", "Sender", "Subject",
                    "Recipient", "ReportID", "DeviceGroupHierarchyLevel1", "DeviceGroupHierarchyLevel2",
                    "DeviceGroupHierarchyLevel3", "DeviceGroupHierarchyLevel4", "VirtualSystemName",
                    "DeviceName", "FUTUREUSE", "SourceVMUUID", "DestinationVMUUID", "HTTPMethod",
                    "TunnelID/IMSI", "MonitorTag/IMEI", "ParentSessionID", "ParentStartTime", "TunnelType",
                    "ThreatCategory", "ContentVersion", "FUTUREUSE", "SCTPAssociationID", "PayloadProtocolID",
                    "HTTPHeaders", "URLCategoryList", "UUIDforrule", "HTTP/2Connection"
                ]
            }

```

Output in ElasticSearch showing SerialNumber mapped to Type  
 ![image](https://us1.discourse-cdn.com/elastic/original/3X/4/c/4cdbefa73b8a06b03a2c6df610487985ed6f412d.png)

---

<div class="post-metadata">

**Author:** ![ylasri](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/ylasri/32/86120_2.png) [@ylasri](https://discuss.elastic.co/u/ylasri)\
**Post date:** [September 24, 2020, 5:31pm UTC](https://discuss.elastic.co/t/parsing-issues-with-logstash-csv-plugin/249856/2 "2020-09-24T17:31:09Z")

</div>

Are you refering [Traffic](https://docs.paloaltonetworks.com/pan-os/9-0/pan-os-admin/monitoring/use-syslog-for-monitoring/syslog-field-descriptions/traffic-log-fields.html#idbe18d2d4-9eb8-4966-bec8-df3a6de70e66) or [Threat](https://docs.paloaltonetworks.com/pan-os/9-0/pan-os-admin/monitoring/use-syslog-for-monitoring/syslog-field-descriptions/threat-log-fields.html#id83052cb2-4798-4f9c-abf8-e0b929ce7a3b) logs ?  
The link is about Traffic Log

```
if ([message] =~ /TRAFFIC/) {
            csv {
                source => "message"
                columns => [ 
                    "FUTURE_USE", "ReceiveTime", "SerialNumber", "Type", "Threat_ContentType", "FUTURE_USE",
                    "GeneratedTime", "SourceIP", "DestinationIP", "NATSourceIP", "NATDestinationIP", "RuleName",
                    "SourceUser", "DestinationUser", "Application", "VirtualSystem", "SourceZone", "DestinationZone",
                    "InboundInterface", "OutboundInterface", "LogAction", "FUTURE_USE", "SessionID",
                    "RepeatCount", "SourcePort", "DestinationPort", "NATSourcePort", "NATDestinationPort", "Flags",
                    "Protocol", "Action", "Bytes", "BytesSent", "BytesReceived", "Packets", "StartTime", "ElapsedTime",
                    "Category", "FUTURE_USE", "SequenceNumber", "ActionFlags", "SourceLocation", 
                    "DestinationLocation", "FUTURE_USE", "PacketsSent", "PacketsReceived", "SessionEndReason",
                    "DeviceGroupHierarchyLevel1", "DeviceGroupHierarchyLevel2", "DeviceGroupHierarchyLevel3",
                    "DeviceGroupHierarchyLevel4", "VirtualSystemName", "DeviceName", "ActionSource", "SourceVMUUID",
                    "DestinationVMUUID", "TunnelID_IMSI", "MonitorTag_IMEI", "ParentSessionID", "ParentStartTime",
                    "TunnelType", "SCTPAssociationID", "SCTPChunks", "SCTPChunksSent", "SCTPChunksReceived"
                ]
            }
        }

        else if ([message] =~ /THREAT/) {
            csv {
                source => "message"
                columns => [ 
                    "FUTURE_USE", "ReceiveTime", "SerialNumber", "Type", "Threat_ContentType", "FUTURE_USE",
                    "GeneratedTime", "SourceIP", "DestinationIP", "NATSourceIP", "NATDestinationIP", "RuleName", 
                    "SourceUser", "DestinationUser", "Application", "VirtualSystem", "SourceZone", "DestinationZone",
                    "InboundInterface", "OutboundInterface", "LogAction", "FUTURE_USE", "SessionID",
                    "RepeatCount", "SourcePort", "DestinationPort", "NATSourcePort", "NATDestinationPort", "Flags",
                    "Protocol", "Action", "URL_Filename", "ThreatID", "Category", "Severity", "Direction",
                    "SequenceNumber", "ActionFlags", "SourceLocation", "DestinationLocation", "FUTURE_USE", 
                    "ContentType", "PCAP_ID", "FileDigest", "Cloud", "URLIndex", "UserAgent", "FileType",
                    "X-Forwarded-For", "Referer", "Sender", "Subject", "Recipient", "ReportID",
                    "DeviceGroupHierarchyLevel1", "DeviceGroupHierarchyLevel2", "DeviceGroupHierarchyLevel3",
                    "DeviceGroupHierarchyLevel4", "VirtualSystemName", "DeviceName", "FUTURE_USE", "SourceVMUUID",
                    "DestinationVMUUID", "HTTPMethod", "TunnelID_IMSI", "MonitorTag_IMEI", "ParentSessionID", 
                    "ParentStartTime", "TunnelType", "ThreatCategory", "ContentVersion", "FUTURE_USE" , 
                    "SCTPAssociationID", "PayloadProtocolID", "HTTPHeaders" 
                ]
            }
        }
```

---

<div class="post-metadata">

**Author:** ![petehalatsis](https://avatars.discourse-cdn.com/v4/letter/p/bbce88/32.png) [@petehalatsis](https://discuss.elastic.co/u/petehalatsis)\
**Post date:** [September 24, 2020, 5:45pm UTC](https://discuss.elastic.co/t/parsing-issues-with-logstash-csv-plugin/249856/3 "2020-09-24T17:45:43Z")

</div>

Sorry about that. I corrected the link to the correct doc that I was referring to.

---

<div class="post-metadata">

**Author:** ![petehalatsis](https://avatars.discourse-cdn.com/v4/letter/p/bbce88/32.png) [@petehalatsis](https://discuss.elastic.co/u/petehalatsis)\
**Post date:** [September 25, 2020, 8:13pm UTC](https://discuss.elastic.co/t/parsing-issues-with-logstash-csv-plugin/249856/4 "2020-09-25T20:13:19Z")

</div>

I proved Einstein wrong. I did the same thing over and over and eventually got different results. My logs are coming in fine now. I removed the first column (didn't need it anyways), deleted my index (test data anyways), and recreated.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [October 23, 2020, 8:13pm UTC](https://discuss.elastic.co/t/parsing-issues-with-logstash-csv-plugin/249856/5 "2020-10-23T20:13:25Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.