I am trying to get syslog from Palo Alto to ElasticSearch. I found that when I use a filter I do not get any line feed and carriage returns and no data gets sent to Elastic Search.
This code does not send any data to ElasticSearch
input {
udp {
port => 514
type => syslog
}
}
filter {
if [type] == "syslog" {
csv {
columns => ["FUTURE_USE"," Receive Time"," Serial Number"," Type"," Subtype"," FUTURE_USE"," Generated Time"," Source IP"," Destination IP"," NAT Source IP"," NAT Destination IP"," Rule Name"," Source User"," Destination User"," Application"," Virtual System"," Source Zone"," Destination Zone"," Ingress Interface"," Egress Interface"," Log Forwarding Profile"," FUTURE_USE"," Session ID"," Repeat Count"," Source Port"," Destination Port"," NAT Source Port"," NAT Destination Port"," Flags"," Protocol"," Action"," Miscellaneous"," Threat ID"," Category"," Severity"," Direction"," Sequence Number"," Action Flags"," Source Location"," Destination Location"," FUTURE_USE"," Content Type"," PCAP_id"," Filedigest"," Cloud"," FUTURE_USE"," User Agent * "," File Type * "," X-Forwarded-For * "," Referer * "," Sender * "," Subject * "," Recipient * "," Report ID * "]
}
}
output {
elasticsearch {
hosts => "localhost:9200"
}
stdout { codec => rubydebug }
}
************* results ***************
"FUTURE_USE"=>"<14>May 19 14:22:30 PA-3050-Primary.uhsh.uhs.or
g 1", " Receive Time"=>"2016/05/19 14:22:30", " Serial Number"=>"001701007562",
" Type"=>"TRAFFIC", " Subtype"=>"end", " FUTURE_USE"=>"1", " Generated Time"=>"2
016/05/19 14:22:30", " Source IP"=>"192.168.23.43", " Destination IP"=>"209.244.
0.3", " NAT Source IP"=>"0.0.0.0", " NAT Destination IP"=>"0.0.0.0", " Rule Name
"=>"UHS_Secure", " Source User"=>nil, " Destination User"=>nil, " Application"=>
"dns", " Virtual System"=>"vsys1", " Source Zone"=>"UHS_Secure_Int", " Destinati
on Zone"=>"UHS_Secure_Ext", " Ingress Interface"=>"ethernet1/2", " Egress Interf
ace"=>"ethernet1/1", " Log Forwarding Profile"=>"Syslog", " Session ID"=>"199216
", " Repeat Count"=>"1", " Source Port"=>"61257", " Destination Port"=>"53", " N
AT Source Port"=>"0", " NAT Destination Port"=>"0", " Flags"=>"0x19", " Protocol
"=>"udp", " Action"=>"allow", " Miscellaneous"=>"208", " Threat ID"=>"88", " Cat
egory"=>"120", " Severity"=>"2", " Direction"=>"2016/05/19 14:22:00", " Sequence
Number"=>"30", " Action Flags"=>"any", " Source Location"=>"0", " Destination L
ocation"=>"8489029195", " Content Type"=>"192.168.0.0-192.168.255.255", " PCAP_i
d"=>"US", " Filedigest"=>"0", " Cloud"=>"1", " User Agent * "=>"aged-out\u0000"}
, " NAT Source IP"], "[ NAT Destination IP]"=>[{"message"=>"<14>May 19 14:22:30
PA-3050-Primary.uhsh.uhs.org 1,2016/05/19 14:22:30,001701007562,TRAFFIC,end,1,20
16/05/19 14:22:30,192.168.23.43,209.244.0.3,0.0.0.0,0.0.0.0,UHS_Secure,,,dns,vsy
s1,UHS_Secure_Int,UHS_Secure_Ext,ethernet1/2,ethernet1/1,Syslog,2016/05/19 14:22
:30,199216,1,61257,53,0,0,0x19,udp,allow,208,88,120,2,2016/05/19 14:22:00,30,any
,0,8489029195,0x0,192.168.0.0-192.168.255.255,US,0,1,1,aged-out\u0000", "@versio
n"=>"1", "@timestamp"=>"2016-05-19T18:23:25.641Z", "type"=>"syslog", "host"=>"10
.20.23.7", "FUTURE_USE"=>"<14>May 19 14:22:30 PA-3050-Primary.uhsh.uhs.org 1", "
Receive Time"=>"2016/05/19 14:22:30", " Serial Number"=>"001701007562", " Type"
If I don't use a filter I get:
input {
udp {
port => 514
type => syslog
}
}
output {
elasticsearch {
hosts => "localhost:9200"
}
stdout { codec => rubydebug }
}
********************* results sent to ElasticSearch ************************
{
"message" => "<14>May 19 14:30:34 PA-3050-Primary.uhsh.uhs.org 1,2016/05/
19 14:30:34,001701007562,TRAFFIC,end,1,2016/05/19 14:30:34,172.25.25.161,10.23.1
.46,0.0.0.0,0.0.0.0,UHS_Secure,,,incomplete,vsys1,UHS_Secure_Ext,UHS_Secure_Int,
ethernet1/1,ethernet1/2,Syslog,2016/05/19 14:30:34,361589,1,23635,80,0,0,0x1c,tc
p,allow,300,120,180,6,2016/05/19 14:30:17,15,any,0,8489308808,0x0,172.16.0.0-172
.31.255.255,10.0.0.0-10.255.255.255,0,3,3,tcp-fin\u0000",
"@version" => "1",
"@timestamp" => "2016-05-19T18:31:27.638Z",
"type" => "syslog",
"host" => "10.20.23.7"
}
And that works and data gets sent to ElasticSearch.... but I only get these 4 fields not all the available fields.
Please help.... spent many hrs with no success.
Thanks.... Tom