Hi All,
I have created a logstash pipeline to read the network syslog (RFC5424) data as mentioned below, However I don't see any output while running the pipeline.
logstash pipeline
input {
#tcp {
# host => "0.0.0.0"
# port => 8001
# }
syslog {
port => 8001
codec => cef
syslog_field => "syslog"
grok_pattern => "<%{POSINT:priority}>%{SYSLOGTIMESTAMP:timestamp}%{SYSLOG5424PRI}%{GREEDYDATA:syslog5424_sd}"
}
}
filter {
}
output {
stdout {}
}
pipeline execution logs
[INFO ] 2022-10-21 10:59:00.211 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"}
[INFO ] 2022-10-21 10:59:00.241 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[INFO ] 2022-10-21 10:59:00.275 [Ruby-0-Thread-15: :1] syslog - Starting syslog udp listener {:address=>"0.0.0.0:9004"}
[INFO ] 2022-10-21 10:59:00.306 [Ruby-0-Thread-17: :1] syslog - Starting syslog tcp listener {:address=>"0.0.0.0:9004"}
[INFO ] 2022-10-21 10:59:16.092 [Ruby-0-Thread-19: :1] syslog - new connection {:client=>"*******:*****"}
[INFO ] 2022-10-21 11:00:36.511 [Ruby-0-Thread-20: :1] syslog - new connection {:client=>"*******:*****"}
Sample data:
Rios
October 21, 2022, 11:19am
2
This is not similar to CEF format:Sep 19 08:26:10 host CEF:0|Security|threatmanager|1.0|100|worm successfully stopped|10|src=10.0.0.1 dst=2.1.2.2 spt=1232
syslog {
port => 8001
syslog_field => "syslog"
}
}
Your sample is most likely for kv filter.
Rios:
without codec and grok
Thanks Rios for the reply, I removed grok_pattern , codec and tested again, but no luck. Note: sample data was copied from TCP input plugin result.
leandrojmp
October 21, 2022, 1:01pm
4
This is not CEF nor it follows the syslog pattern that the syslog input expectes.
If you are receiving only the Fortigate logs in this port, you should switch to the tcp input, if you are receiving anything else you should configure your fortinet device to send to another port that will listen with the tcp input.
To parse this you just need these filters:
filter {
dissect {
mapping => {
"message" => "<%{}>%{kvmsg}"
}
}
mutate {
strip => ["kvmsg"]
}
kv {
source => "kvmsg"
target => "[fortinet][firewall]"
}
}
This will parse your message and put the fields under fortinet.firewall, example fortinet.firewall.srcip.
leandrojmp:
srcip
Thanks Leandrojmp, I updated the logstash script as assisted; however it is still printing the similar data as returned by TCP {i.e. <188>date.......}
updated code
input {
tcp {
host => "0.0.0.0"
port => 8001
}
}
filter {
dissect {
mapping => {
"message" => "<%{}>%{kvmsg}"
}
}
mutate {
strip => ["kvmsg"]
}
kv {
source => "message"
target => "[fortinet][firewall]"
}
}
output {
stdout { codec => line { format => "%{[fortinet][firewall][srcip]}" }}
}
leandrojmp
October 21, 2022, 1:58pm
6
Sorry, in the source for the kv filter you should use the kvmsg field.
Change the pipeline and share your output.
Remove this, there is no need for it, it will make you output this field only, if it exists.
@ leandrojmp
leandrojmp
October 21, 2022, 5:00pm
8
You need to share your output.
@ leandrojmp Please find the output below; note: this is just part of data, as I cannot copy and share the entire.
<188>date=2022-09-21 time=10:38:43 devname=\\\"test\\\" devid=\\\"test8918585544546\\\" eventtime=166634345346307470 tz=\\\"+0000\\\" logid=\\\"0000000000\\\" type=\\\"traffic\\\" subtype=\\\"forward\\\" level=\\\"warning\\\" vd=\\\"root\\\" srcip=00.0.00.000 srcport=00000 srcintf=\\\"VLAN0000\\" srcintfrole=\\\"lbv\\\" dstip=000.000.000.000 dstport=00 dstintf=\\\"test\\\" dstintfrole=\\\"tst\\\" srcuuid=\\\"0f0000ea-0ac0-01e0-beb0-e0000d00febe\\\" dstuuid=\\\"0f00001ac0-01e0-bjb0-j0000d00febe\\\" srccountry=\\\"Reserved\\\" dstcountry=\\\"Reserved\\\" sessionid=00000000 proto=0 action=\\\"ip-conn\\\" policyid=00 policytype=\\\"policy\\\" poluuid=\\\"00b000-0000-00ea-0000-f00000a00000\\\" policyname=\\\"LAN > WAN - test\\\" centralnatid=0 service=\\\"HTTP\\\" appcat=\\\"unscanned\\\" crscore=0 craction=00000 crlevel=\\\"low\\\" srchwvendor=\\\"test\\\" devtype=\\\"Network\\\" srcfamily=\\\"Router\\\" osname=\\\"test\\\" mastersrcmac=\\\"f0:b0:50:0d:a0:c0\\\" srcmac=\\\"f0:b0:00:80d:a0:c0\\\" srcserver=0000<190>date=2022-09-21 time=10:38:43 devname=\\\"Testdev\\\" devid=\\\"Test111\\\" eventtime=1111111111111111111 tz=\\\"+1111\\\" logid=\\\"1111111111\\\" type=\\\"utm\\\" subtype=\\\"app-ctrl\\\" eventtype=\\\"signature\\\" level=\\\"information\\\" vd=\\\"root\\\" appid=11111 srcip=11.11.1.111 dstip=111.111.111.111 srcport=11111 dstport=111 srcintf=\\\"V6tv1111\\\" srcintfrole=\\\"1bv\\\" dstintf=\\\"testdest\\\" dstintfrole=\\\"hfn\\\" proto=1 service=\\\"SSL\\\" direction=\\\"outgoing\\\" policyid=11 sessionid=11111111 applist=\\\"default\\\" action=\\\"pass\\\" appcat=\\\"Update\\\" app=\\\"test.gs.Update\\\" hostname=\\\"test.test.com\\\" incidentserialno=111111111 url=\\\"/\\\" msg=\\\"Update: test.gg.Update,\\\" apprisk=\\\"low\\\" scertcname=\\\"test.test.com\\\" scertissuer=\\\"TLS RSA SHA111 1111 CA1\\\"111 <189>date=2022-09-21 time=10:38:42 devname=\\\"test\\\" devid=\\\"test8585\\\" eventtime=1666434342179976 tz=\\\"+0110\\\" logid=\\\"0000000000\\\" type=\\\"traffic\\\" subtype=\\\"forward\\\" level=\\\"test\\\" vd=\\\"root\\\" srcip=00.00.0.00 srcname=\\\"LAPTOP-test\\\" srcport=11111 srcintf=\\\"VLAN111\\\" srcintfrole=\\\"0bv\\\" dstip=11.111.11.11 dstport=111 dstintf=\\\"test\\\" dstintfrole=\\\"tst\\\" srcuuid=\\\"1f1111ea-1ac1-11e1-beb1-e1111d11febe\\\" dstuuid=\\\"1f1111ea-1ac1-11e1-beb1-e1111d11febe\\\" srccountry=\\\"Reserved\\\" dstcountry=\\\"TEST\\\" sessionid=11111111 proto=1 action=\\\"accept\\\" policyid=11 policytype=\\\"policy\\\" poluuid=\\\"1f1f1111-1111-11ea-1111-11111db111\\\" policyname=\\\"test > WAN - Allow All test\\\" centralnatid=0 service=\\\"HTTPS\\\" trandisp=\\\"test\\\" transip=000.00.000.00 transport=11111 duration=111 sentbyte=111111 rcvdbyte=111111 sentpkt=111 rcvdpkt=111 appcat=\\\"unscanned\\\" sentdelta=111 rcvddelta=111 osname=\\\"test\\\" srcswversion=\\\"101 / 2018\\\" mastersrcmac=\\\"00:e0:0a:0f:ad:0d\\\" srcmac=\\\"00:e0:0a:0f:ad:0d\\\" srcserver=00000
leandrojmp
October 21, 2022, 7:52pm
10
You need to share the output in the json format, it is not possible to understand what is happening without seeing the entire json output to check for errors.
The pipeline is correct and should parse it, without seeing what is your output is not possible to help further.
Badger
October 21, 2022, 11:12pm
11
When I run with that input and the filter/output you showed I get
00.0.00.000,11.11.1.111,00.00.0.00
@leandrojmp With fortinet syslog format = default , I was getting the above output. Now we have changed the format in fortinet to rfc5425 and I can see ouput in Json as mentioned below - Note: Now I am trying to implement the solution mentioned in Logstash and RFC5424 — RFC5424 logging handler 1.4.3 documentation (rfc5424-logging-handler.readthedocs.io)
},
"message" => "<189>1 2022-10-24T12:07:57Z Test - - - - eventtime=1666602477071090516 tz=\"+1111\" logid=\"0117011111\" type=\"utm\" subtype=\"webfilter\" eventtype=\"test_allow\" level=\"notice\" vd=\"root\" policyid=11 sessionid=111111111 srcip=11.1.11.111 srcport=11111 srcintf=\"test1111\" srcintfrole=\"ebd\" dstip=11.11.111.11 dstport=111 dstintf=\"test\" dstintfrole=\"wan\" proto=1 service=\"HTTPS\" hostname=\"test.org\" profile=\"monitor-all\" action=\"passthrough\" reqtype=\"direct\" url=\"https://test.org/\" sentbyte=111 rcvdbyte=1 direction=\"outgoing\" msg=\"tet msg\" method=\"domain\" cat=11 catdesc=\"test\""
}
{
"tags" => [
[0] "_grokparsefailure_sysloginput"
],
"service" => {
"type" => "system"
},
"@version" => "1",
"@timestamp" => 2022-10-24T12:07:58.269123Z,
"host" => {
"ip" => "11.11.1.11"
},
"log" => {
"syslog" => {
"facility" => {
"code" => 0,
"name" => "kernel"
},
"priority" => 0,
"severity" => {
"code" => 0,
"name" => "Emergency"
}
}
},
"event" => {
"original" => nil
},
"message" => "<190>1 2022-10-24T11:07:57Z test - - - - eventtime=1111102477071520338 tz=\"+1111\" logid=\"1111128111\" type=\"utm\" subtype=\"test\" eventtype=\"signature\" level=\"information\" vd=\"root\" appid=11111 srcip=11.1.11.11 dstip=20.61.97.195 srcport=11110 dstport=111 srcintf=\"test1111\" srcintfrole=\"bvd\" dstintf=\"test\" dstintfrole=\"wan\" proto=1 service=\"SSL\" direction=\"outgoing\" policyid=11sessionid=111111111 applist=\"default\" action=\"pass\" appcat=\"test.Service\" app=\"SSL\" hostname=\"test.io\" incidentserialno=1111111 url=\"/\" msg=\"test: SSL,\" apprisk=\"elevated\" scertcname=\"*.test.io\" scertissuer=\"test Issuing CA 06\""
}
{
"tags" => [
[0] "_grokparsefailure_sysloginput"
],
"service" => {
"type" => "system"
},
"@version" => "1",
"@timestamp" => 2022-10-24T11:07:58.269527Z,
"host" => {
"ip" => "11.11.1.11"
},
"log" => {
"syslog" => {
"facility" => {
"code" => 0,
"name" => "kernel"
},
"priority" => 0,
"severity" => {
"code" => 0,
"name" => "Emergency"
}
}
},
"event" => {
"original" => nil
},
```
leandrojmp
October 24, 2022, 1:37pm
13
I'm not sure what is the issue, you are still using the syslog input?
You should use the tcp input to receive the fortinet logs and parse it with dissect and kv.
The syslog input is pretty limited, for example it does not support the rfc5425 you are using, it only supports syslog messages that follows RFC3164.
To parse the Fortinet logs you need to configure the format to default in the device and use the TCP input with the example pipeline I shared in a previous answer.
I was using the tcp input only {as assisted by you on Friday} along with the "default" syslog format; with that setup, I was not getting the Json output.
leandrojmp
October 24, 2022, 1:55pm
15
Are you sure that your device is sending log using TCP and not UDP?
Check which protocol your device is sending log and configure it in the input.
The syslog input will not work in your case you need the tcp or udp input to receive the logs.
Try to open the same port both in TCP and UDP.
I already tested traffic using tcpdump -n port 8001 and I can see the events.
leandrojmp
October 24, 2022, 2:23pm
17
I'm not sure what is the issue if you are not receiving logs with the TCP input.
Try the following pipeline in Logstash:
input {
tcp {
port => 8001
}
}
output {
stdout {}
}
Check that your fortinet device is configured to send logs to your logstash server on port 8001 using TCP and the default format, if this is right you should receive the raw logs at least.
Thanks a lot @leandrojmp .
Also, now I am seeing events being inserted in elastic by kibana integration over udp, I will skip logstash pipeline {as I was working on prod env and cannot use it for testing now).
Thanks a lot for your support, I would really like to appreciate your efforts to assist me throughout, I owe you a good coffee
},
"message" => "<189>1 2022-10-24T12:07:57Z Test - - - - eventtime=1666602477071090516 tz=\"+1111\" logid=\"0117011111\" type=\"utm\" subtype=\"webfilter\" eventtype=\"test_allow\" level=\"notice\" vd=\"root\" policyid=11 sessionid=111111111 srcip=11.1.11.111 srcport=11111 srcintf=\"test1111\" srcintfrole=\"ebd\" dstip=11.11.111.11 dstport=111 dstintf=\"test\" dstintfrole=\"wan\" proto=1 service=\"HTTPS\" hostname=\"test.org\" profile=\"monitor-all\" action=\"passthrough\" reqtype=\"direct\" url=\"https://test.org/\" sentbyte=111 rcvdbyte=1 direction=\"outgoing\" msg=\"tet msg\" method=\"domain\" cat=11 catdesc=\"test\""
}
{
"@timestamp" => 2022-10-24T14:38:52.986895Z,
"@version" => "1",
"event" => {
"original" => "<189>1 2022-10-24T12:07:57Z Test - - - - eventtime=1666602477071090516 tz=\"+1111\" logid=\"0117011111\" type=\"utm\" subtype=\"webfilter\" eventtype=\"test_allow\" level=\"notice\" vd=\"root\" policyid=11 sessionid=111111111 srcip=11.1.11.111 srcport=11111 srcintf=\"test1111\" srcintfrole=\"ebd\" dstip=11.11.111.11 dstport=111 dstintf=\"test\" dstintfrole=\"wan\" proto=1 service=\"HTTPS\" hostname=\"test.org\" profile=\"monitor-all\" action=\"passthrough\" reqtype=\"direct\" url=\"https://test.org/\" sentbyte=111 rcvdbyte=1 direction=\"outgoing\" msg=\"tet msg\" method=\"domain\" cat=11 catdesc=\"test\""
}
system
November 21, 2022, 2:59pm
19
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.
