I'm attempting to parse Versa Analytics Events. The log stream from the SD-Wan devices will have several event types in the over all stream. E.g. URL event logs, FW event logs, IDS event logs etc. My config is as follows and there is something that I'm missing becasue config causes logstash to fail:
filter {
if [logname] == "urlfLog" {
grok {
patterns_dir => ["/etc/logstash/patterns"]
match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{WORD:logname}, applianceName=%{textDef:applianceName}, tenantName=%{textDef:tenantName}, flowId=%{NONNEGINT:flowId}, flowCookie=%{NONNEGINT:flowCookie}, vsnId=%{INT:vsnId}, applianceId=%{NONNEGINT:applianceId}, tenantId=%{NONNEGINT:tenantId}, urlReputation=%{textDef:urlReputation}, urlCategory=%{textDef:urlCategory}, httpUrl=%{urlDef:httpUrl}, urlfProfile=%{textDef:urlfProfile}, urlfAction=%{textDef:urlfAction}, urlfActionMessage=%{textDef:urlfActionMessage}, sourceIPv4Address=%{IPV4:sourceIPv4Address}, destinationIPv4Address=%{IPV4:destinationIPv4Address}, sourceTransportPort=%{NONNEGINT:sourceTransportPort}, destinationTransportPort=%{NONNEGINT:destinationTransportPort}" }
}
} else if [logname] == "accessLog" {
grok {
patterns_dir => ["/etc/logstash/patterns"]
match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{WORD:logname}, applianceName=%{textDef:applianceName}, tenantName=%{textDef:tenantName}, flowId=%{NONNEGINT:flowId}, flowCookie=%{NONNEGINT:flowCookie}, flowStartMilliseconds=%{NONNEGINT:flowStartMilliseconds}, flowEndMilliseconds=%{NONNEGINT:flowENDMilliseconds}, sentOctets=%{NONNEGINT:sentOctets}, sentPackets=%{NONNEGINT:sentPackets}, recvdOctets=%{NONNEGINT:recvdOctets}, recvdPackets=%{NONNEGINT:recvdPackets}, appId=%{NONNEGINT:appId}, eventType=%{textDef:eventType}, tenantId=%{NONNEGINT:tenantId}, urlCategory=%{textDef:urlCategory}, action=%{textDef:action}, vsnId=%{NONNEGINT:vsnId}, applianceId=%{NONNEGINT:applianceId}, appRisk=%{NONNEGINT:appRisk}, appProductivity=%{NONNEGINT:appProductivity}, appIdStr=%{textDef:appIdStr}, appFamily=%{textDef:appFamily}, appSubFamily=%{textDef:appSubFamily}, rule=%{textDef:rule}, forwardForwardingClass=%{textDef:forwardForwardingClass}, reverseForwardingClass=%{textDef:reverseForwardingClass}, host=%{textDef:host}, deviceKey=%{NONNEGINT:deviceKey}, deviceName=%{textDef:deviceName}, sourceIPv4Address=%{IPV4:sourceIPv4Address}, destinationIPv4Address=%{IPV4:destinationIPv4Address}, sourceTransportPort=%{NONNEGINT:sourceTransportPort}, destinationTransportPort=%{NONNEGINT:destinationTransportPort}" }
}
} else drop{}
}
Logstash log file has the following -
2020-10-09T16:01:01,447][ERROR][logstash.agent ] Failed to execute action {:action=>LogStash::PipelineAction::Create/pipeline_id:main, :exception=>"LogStash::ConfigurationError", :message=>"Expected one of [ \t\r\n], "#", "if", "{" at line 20, column 9 (byte 2394) after filter {\n\tif [logname] =="urlfLog" {\n\t grok {\n\t\t patterns_dir => ["/etc/logstash/patterns"]\n\t\t match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{WORD:logname}, applianceName=%{textDef:applianceName}, tenantName=%{textDef:tenantName}, flowId=%{NONNEGINT:flowId}, flowCookie=%{NONNEGINT:flowCookie}, vsnId=%{INT:vsnId}, applianceId=%{NONNEGINT:applianceId}, tenantId=%{NONNEGINT:tenantId}, urlReputation=%{textDef:urlReputation}, urlCategory=%{textDef:urlCategory}, httpUrl=%{urlDef:httpUrl}, urlfProfile=%{textDef:urlfProfile}, urlfAction=%{textDef:urlfAction}, urlfActionMessage=%{textDef:urlfActionMessage}, sourceIPv4Address=%{IPV4:sourceIPv4Address}, destinationIPv4Address=%{IPV4:destinationIPv4Address}, sourceTransportPort=%{NONNEGINT:sourceTransportPort}, destinationTransportPort=%{NONNEGINT:destinationTransportPort}" }\n\t\t\t }\n\t} else if [logname] == "accessLog" {\n\t grok {\n\t\t patterns_dir => ["/etc/logstash/patterns"]\n\t\t match => { "message" => "%{TIMESTAMP_ISO8601:timestamp} %{WORD:logname}, applianceName=%{textDef:applianceName}, tenantName=%{textDef:tenantName}, flowId=%{NONNEGINT:flowId}, flowCookie=%{NONNEGINT:flowCookie}, flowStartMilliseconds=%{NONNEGINT:flowStartMilliseconds}, flowEndMilliseconds=%{NONNEGINT:flowENDMilliseconds}, sentOctets=%{NONNEGINT:sentOctets}, sentPackets=%{NONNEGINT:sentPackets}, recvdOctets=%{NONNEGINT:recvdOctets}, recvdPackets=%{NONNEGINT:recvdPackets}, appId=%{NONNEGINT:appId}, eventType=%{textDef:eventType}, tenantId=%{NONNEGINT:tenantId}, urlCategory=%{textDef:urlCategory}, action=%{textDef:action}, vsnId=%{NONNEGINT:vsnId}, applianceId=%{NONNEGINT:applianceId}, appRisk=%{NONNEGINT:appRisk}, appProductivity=%{NONNEGINT:appProductivity}, appIdStr=%{textDef:appIdStr}, appFamily=%{textDef:appFamily}, appSubFamily=%{textDef:appSubFamily}, rule=%{textDef:rule}, forwardForwardingClass=%{textDef:forwardForwardingClass}, reverseForwardingClass=%{textDef:reverseForwardingClass}, host=%{textDef:host}, deviceKey=%{NONNEGINT:deviceKey}, deviceName=%{textDef:deviceName}, sourceIPv4Address=%{IPV4:sourceIPv4Address}, destinationIPv4Address=%{IPV4:destinationIPv4Address}, sourceTransportPort=%{NONNEGINT:sourceTransportPort}, destinationTransportPort=%{NONNEGINT:destinationTransportPort}" }\n\t\t } \n\t} else ", :backtrace=>["/usr/share/logstash/logstash-core/lib/logstash/compiler.rb:32:in compile_imperative'", "org/logstash/execution/AbstractPipelineExt.java:183:in initialize'", "org/logstash/execution/JavaBasePipelineExt.java:69:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/java_pipeline.rb:44:in initialize'", "/usr/share/logstash/logstash-core/lib/logstash/pipeline_action/create.rb:52:in execute'", "/usr/share/logstash/logstash-core/lib/logstash/agent.rb:357:in block in converge_state'"]}
[2020-10-09T16:01:01,907][INFO ][logstash.agent ] Successfully started Logstash API endpoint {:port=>9600}
[2020-10-09T16:01:06,811][INFO ][logstash.runner ] Logstash shut down.
[2020-10-09T16:01:06,841][ERROR][org.logstash.Logstash ] java.lang.IllegalStateException: Logstash stopped processing because of an error: (SystemExit) exit
Thoughts and comments are appreciated
TimW