Parsing issues with Logstash CSV plugin

Elastic Stack Logstash
1

I have been beating my head on this for a while. My syslog is hitting my if statement and using the columns I have below but they are off by one. I tried adding/removing the first column to account for the shift but it doubles my error somehow. Is there something silly that I am doing wrong here?

(Column reference: https://docs.paloaltonetworks.com/pan-os/9-0/pan-os-admin/monitoring/use-syslog-for-monitoring/syslog-field-descriptions/threat-log-fields.html)

if ([message] =~ /THREAT/) {
            csv {
                source => "message"
                columns => [ 
                    "FUTUREUSE", "ReceiveTime", "SerialNumber", "Type", "Threat/ContentType", "FUTUREUSE",
                    "GeneratedTime", "SourceAddress", "DestinationIP", "NATSourceIP", "NATDestinationIP",
                    "RuleName", "SourceUser", "DestinationUser", "Application", "VirtualSystem", "SourceZone",
                    "DestinationZone", "InboundInterface", "OutboundInterface", "LogAction", "FUTUREUSE",
                    "SessionID", "RepeatCount", "SourcePort", "DestinationPort", "NATSourcePort",
                    "NATDestinationPort", "Flags", "Protocol", "Action", "URL/Filename", "ThreatID",
                    "Category", "Severity", "Direction", "SequenceNumber", "ActionFlags", "SourceLocation",
                    "DestinationLocation", "FUTUREUSE", "ContentType", "PCAP_ID", "FileDigest", "Cloud",
                    "URLIndex", "User Agent", "FileType", "X-Forwarded-For", "Referer", "Sender", "Subject",
                    "Recipient", "ReportID", "DeviceGroupHierarchyLevel1", "DeviceGroupHierarchyLevel2",
                    "DeviceGroupHierarchyLevel3", "DeviceGroupHierarchyLevel4", "VirtualSystemName",
                    "DeviceName", "FUTUREUSE", "SourceVMUUID", "DestinationVMUUID", "HTTPMethod",
                    "TunnelID/IMSI", "MonitorTag/IMEI", "ParentSessionID", "ParentStartTime", "TunnelType",
                    "ThreatCategory", "ContentVersion", "FUTUREUSE", "SCTPAssociationID", "PayloadProtocolID",
                    "HTTPHeaders", "URLCategoryList", "UUIDforrule", "HTTP/2Connection"
                ]
            }

Output in ElasticSearch showing SerialNumber mapped to Type
image

2

Are you refering Traffic or Threat logs ?
The link is about Traffic Log

if ([message] =~ /TRAFFIC/) {
            csv {
                source => "message"
                columns => [ 
                    "FUTURE_USE", "ReceiveTime", "SerialNumber", "Type", "Threat_ContentType", "FUTURE_USE",
                    "GeneratedTime", "SourceIP", "DestinationIP", "NATSourceIP", "NATDestinationIP", "RuleName",
                    "SourceUser", "DestinationUser", "Application", "VirtualSystem", "SourceZone", "DestinationZone",
                    "InboundInterface", "OutboundInterface", "LogAction", "FUTURE_USE", "SessionID",
                    "RepeatCount", "SourcePort", "DestinationPort", "NATSourcePort", "NATDestinationPort", "Flags",
                    "Protocol", "Action", "Bytes", "BytesSent", "BytesReceived", "Packets", "StartTime", "ElapsedTime",
                    "Category", "FUTURE_USE", "SequenceNumber", "ActionFlags", "SourceLocation", 
                    "DestinationLocation", "FUTURE_USE", "PacketsSent", "PacketsReceived", "SessionEndReason",
                    "DeviceGroupHierarchyLevel1", "DeviceGroupHierarchyLevel2", "DeviceGroupHierarchyLevel3",
                    "DeviceGroupHierarchyLevel4", "VirtualSystemName", "DeviceName", "ActionSource", "SourceVMUUID",
                    "DestinationVMUUID", "TunnelID_IMSI", "MonitorTag_IMEI", "ParentSessionID", "ParentStartTime",
                    "TunnelType", "SCTPAssociationID", "SCTPChunks", "SCTPChunksSent", "SCTPChunksReceived"
                ]
            }
        }

        else if ([message] =~ /THREAT/) {
            csv {
                source => "message"
                columns => [ 
                    "FUTURE_USE", "ReceiveTime", "SerialNumber", "Type", "Threat_ContentType", "FUTURE_USE",
                    "GeneratedTime", "SourceIP", "DestinationIP", "NATSourceIP", "NATDestinationIP", "RuleName", 
                    "SourceUser", "DestinationUser", "Application", "VirtualSystem", "SourceZone", "DestinationZone",
                    "InboundInterface", "OutboundInterface", "LogAction", "FUTURE_USE", "SessionID",
                    "RepeatCount", "SourcePort", "DestinationPort", "NATSourcePort", "NATDestinationPort", "Flags",
                    "Protocol", "Action", "URL_Filename", "ThreatID", "Category", "Severity", "Direction",
                    "SequenceNumber", "ActionFlags", "SourceLocation", "DestinationLocation", "FUTURE_USE", 
                    "ContentType", "PCAP_ID", "FileDigest", "Cloud", "URLIndex", "UserAgent", "FileType",
                    "X-Forwarded-For", "Referer", "Sender", "Subject", "Recipient", "ReportID",
                    "DeviceGroupHierarchyLevel1", "DeviceGroupHierarchyLevel2", "DeviceGroupHierarchyLevel3",
                    "DeviceGroupHierarchyLevel4", "VirtualSystemName", "DeviceName", "FUTURE_USE", "SourceVMUUID",
                    "DestinationVMUUID", "HTTPMethod", "TunnelID_IMSI", "MonitorTag_IMEI", "ParentSessionID", 
                    "ParentStartTime", "TunnelType", "ThreatCategory", "ContentVersion", "FUTURE_USE" , 
                    "SCTPAssociationID", "PayloadProtocolID", "HTTPHeaders" 
                ]
            }
        }
3

Sorry about that. I corrected the link to the correct doc that I was referring to.

4

I proved Einstein wrong. I did the same thing over and over and eventually got different results. My logs are coming in fine now. I removed the first column (didn't need it anyways), deleted my index (test data anyways), and recreated.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.