Need help on syslog logstash input plugin

Hi All,

I have created a logstash pipeline to read the network syslog (RFC5424) data as mentioned below, However I don't see any output while running the pipeline.
Can someone please assist me what I am missing.
Note : logs are getting printed out over TCP input plugin though.

logstash pipeline

input {
#tcp {
#     host => "0.0.0.0"
#     port => 8001
#     }
 syslog {
    port => 8001
    codec => cef
    syslog_field => "syslog"
    grok_pattern => "<%{POSINT:priority}>%{SYSLOGTIMESTAMP:timestamp}%{SYSLOG5424PRI}%{GREEDYDATA:syslog5424_sd}"
  }
 }
filter {

}

output {
stdout {}
}

pipeline execution logs

[INFO ] 2022-10-21 10:59:00.211 [[main]-pipeline-manager] javapipeline - Pipeline started {"pipeline.id"=>"main"}
[INFO ] 2022-10-21 10:59:00.241 [Agent thread] agent - Pipelines running {:count=>1, :running_pipelines=>[:main], :non_running_pipelines=>[]}
[INFO ] 2022-10-21 10:59:00.275 [Ruby-0-Thread-15: :1] syslog - Starting syslog udp listener {:address=>"0.0.0.0:9004"}
[INFO ] 2022-10-21 10:59:00.306 [Ruby-0-Thread-17: :1] syslog - Starting syslog tcp listener {:address=>"0.0.0.0:9004"}
[INFO ] 2022-10-21 10:59:16.092 [Ruby-0-Thread-19: :1] syslog - new connection {:client=>"*******:*****"}
[INFO ] 2022-10-21 11:00:36.511 [Ruby-0-Thread-20: :1] syslog - new connection {:client=>"*******:*****"}

Sample data:
<189>date=2022-10-21 time=00:38:42 devname=\"Test01\" devid=\"Test85\" eventtime=1111341523332179976 tz=\"+0200\" logid=\"0000000050\" type=\"traffic\" subtype=\"forward\" level=\"notice\" vd=\"root\" srcip=10.77.1.11 srcname=\"LAP-111111EP\" srcport=11114 srcintf=\"VLAN111\" srcintfrole=\"lre\" dstip=10.111.11.11 dstport=111 dstintf=\"TestNet\" dstintfrole=\"Test\" srcuuid=\"1f1111ea-1ac1-11e1-beb1-e1111d11gebe\" dstuuid=\"1f1111ea-1ac1-11e9-ceb1-e1111o96hebe\" srccountry=\"Reserved\" dstcountry=\"INDIA\" sessionid=11110101 proto=1 action=\"accept\" policyid=11 policytype=\"policy\" poluuid=\"1f1f1111-1111-11ea-1111-1111111dbo11\" policyname=\"Test > WAN - Allow \" centralnatid=0 service=\"HTTPS\" trandisp=\"nqt\" transip=111.11.111.11 transport=11111 duration=111 sentbyte=111111 rcvdbyte=11111 sentpkt=111 rcvdpkt=111 appcat=\"unscanned\" sentdelta=111 rcvddelta=111 osname=\"Windows\" srcswversion=\"11 / 1111\" mastersrcmac=\"11:e1:1a:1f:ad:1d\" srcmac=\"11:e1:1a:1f:ad:1d\" srcserver=11111

This is not similar to CEF format:
Sep 19 08:26:10 host CEF:0|Security|threatmanager|1.0|100|worm successfully stopped|10|src=10.0.0.1 dst=2.1.2.2 spt=1232
Can you change input without codec and grok?

 syslog {
    port => 8001
    syslog_field => "syslog"
  }
 }

Your sample is most likely for kv filter.

Thanks Rios for the reply, I removed grok_pattern , codec and tested again, but no luck. Note: sample data was copied from TCP input plugin result.

This is not CEF nor it follows the syslog pattern that the syslog input expectes.

If you are receiving only the Fortigate logs in this port, you should switch to the tcp input, if you are receiving anything else you should configure your fortinet device to send to another port that will listen with the tcp input.

To parse this you just need these filters:

filter {
    dissect {
        mapping => {
            "message" => "<%{}>%{kvmsg}"
        }
    }
    mutate {
        strip => ["kvmsg"]
    }
    kv {
        source => "kvmsg"
        target => "[fortinet][firewall]"
    }
}

This will parse your message and put the fields under fortinet.firewall, example fortinet.firewall.srcip.

Thanks Leandrojmp, I updated the logstash script as assisted; however it is still printing the similar data as returned by TCP {i.e. <188>date.......}
Am I missing something more?

updated code

input {
tcp {
     host => "0.0.0.0"
     port => 8001
     }
 }

filter {
    dissect {
        mapping => {
            "message" => "<%{}>%{kvmsg}"
        }
    }
    mutate {
        strip => ["kvmsg"]
    }
    kv {
        source => "message"
        target => "[fortinet][firewall]"
    }
}

output {
stdout { codec => line { format => "%{[fortinet][firewall][srcip]}" }}
}

Sorry, in the source for the kv filter you should use the kvmsg field.

Change the pipeline and share your output.

Remove this, there is no need for it, it will make you output this field only, if it exists.

@ leandrojmp
Still no luck getting same output <190>date=2022-10-21 ......
image

You need to share your output.

@ leandrojmp Please find the output below; note: this is just part of data, as I cannot copy and share the entire.

<188>date=2022-09-21 time=10:38:43 devname=\\\"test\\\" devid=\\\"test8918585544546\\\" eventtime=166634345346307470 tz=\\\"+0000\\\" logid=\\\"0000000000\\\" type=\\\"traffic\\\" subtype=\\\"forward\\\" level=\\\"warning\\\" vd=\\\"root\\\" srcip=00.0.00.000 srcport=00000 srcintf=\\\"VLAN0000\\" srcintfrole=\\\"lbv\\\" dstip=000.000.000.000 dstport=00 dstintf=\\\"test\\\" dstintfrole=\\\"tst\\\" srcuuid=\\\"0f0000ea-0ac0-01e0-beb0-e0000d00febe\\\" dstuuid=\\\"0f00001ac0-01e0-bjb0-j0000d00febe\\\" srccountry=\\\"Reserved\\\" dstcountry=\\\"Reserved\\\" sessionid=00000000 proto=0 action=\\\"ip-conn\\\" policyid=00 policytype=\\\"policy\\\" poluuid=\\\"00b000-0000-00ea-0000-f00000a00000\\\" policyname=\\\"LAN > WAN - test\\\" centralnatid=0 service=\\\"HTTP\\\" appcat=\\\"unscanned\\\" crscore=0 craction=00000 crlevel=\\\"low\\\" srchwvendor=\\\"test\\\" devtype=\\\"Network\\\" srcfamily=\\\"Router\\\" osname=\\\"test\\\" mastersrcmac=\\\"f0:b0:50:0d:a0:c0\\\" srcmac=\\\"f0:b0:00:80d:a0:c0\\\" srcserver=0000<190>date=2022-09-21 time=10:38:43 devname=\\\"Testdev\\\" devid=\\\"Test111\\\" eventtime=1111111111111111111 tz=\\\"+1111\\\" logid=\\\"1111111111\\\" type=\\\"utm\\\" subtype=\\\"app-ctrl\\\" eventtype=\\\"signature\\\" level=\\\"information\\\" vd=\\\"root\\\" appid=11111 srcip=11.11.1.111 dstip=111.111.111.111 srcport=11111 dstport=111 srcintf=\\\"V6tv1111\\\" srcintfrole=\\\"1bv\\\" dstintf=\\\"testdest\\\" dstintfrole=\\\"hfn\\\" proto=1 service=\\\"SSL\\\" direction=\\\"outgoing\\\" policyid=11 sessionid=11111111 applist=\\\"default\\\" action=\\\"pass\\\" appcat=\\\"Update\\\" app=\\\"test.gs.Update\\\" hostname=\\\"test.test.com\\\" incidentserialno=111111111 url=\\\"/\\\" msg=\\\"Update: test.gg.Update,\\\" apprisk=\\\"low\\\" scertcname=\\\"test.test.com\\\" scertissuer=\\\"TLS RSA SHA111 1111 CA1\\\"111 <189>date=2022-09-21 time=10:38:42 devname=\\\"test\\\" devid=\\\"test8585\\\" eventtime=1666434342179976 tz=\\\"+0110\\\" logid=\\\"0000000000\\\" type=\\\"traffic\\\" subtype=\\\"forward\\\" level=\\\"test\\\" vd=\\\"root\\\" srcip=00.00.0.00 srcname=\\\"LAPTOP-test\\\" srcport=11111 srcintf=\\\"VLAN111\\\" srcintfrole=\\\"0bv\\\" dstip=11.111.11.11 dstport=111 dstintf=\\\"test\\\" dstintfrole=\\\"tst\\\" srcuuid=\\\"1f1111ea-1ac1-11e1-beb1-e1111d11febe\\\" dstuuid=\\\"1f1111ea-1ac1-11e1-beb1-e1111d11febe\\\" srccountry=\\\"Reserved\\\" dstcountry=\\\"TEST\\\" sessionid=11111111 proto=1 action=\\\"accept\\\" policyid=11 policytype=\\\"policy\\\" poluuid=\\\"1f1f1111-1111-11ea-1111-11111db111\\\" policyname=\\\"test > WAN - Allow All test\\\" centralnatid=0 service=\\\"HTTPS\\\" trandisp=\\\"test\\\" transip=000.00.000.00 transport=11111 duration=111 sentbyte=111111 rcvdbyte=111111 sentpkt=111 rcvdpkt=111 appcat=\\\"unscanned\\\" sentdelta=111 rcvddelta=111 osname=\\\"test\\\" srcswversion=\\\"101 / 2018\\\" mastersrcmac=\\\"00:e0:0a:0f:ad:0d\\\" srcmac=\\\"00:e0:0a:0f:ad:0d\\\" srcserver=00000

You need to share the output in the json format, it is not possible to understand what is happening without seeing the entire json output to check for errors.

The pipeline is correct and should parse it, without seeing what is your output is not possible to help further.

When I run with that input and the filter/output you showed I get

00.0.00.000,11.11.1.111,00.00.0.00

@leandrojmp With fortinet syslog format = default , I was getting the above output. Now we have changed the format in fortinet to rfc5425 and I can see ouput in Json as mentioned below - Note: Now I am trying to implement the solution mentioned in Logstash and RFC5424 — RFC5424 logging handler 1.4.3 documentation (rfc5424-logging-handler.readthedocs.io)

},  
  "message" => "<189>1 2022-10-24T12:07:57Z Test - - - - eventtime=1666602477071090516 tz=\"+1111\" logid=\"0117011111\" type=\"utm\" subtype=\"webfilter\" eventtype=\"test_allow\" level=\"notice\" vd=\"root\" policyid=11 sessionid=111111111 srcip=11.1.11.111 srcport=11111 srcintf=\"test1111\" srcintfrole=\"ebd\" dstip=11.11.111.11 dstport=111 dstintf=\"test\" dstintfrole=\"wan\" proto=1 service=\"HTTPS\" hostname=\"test.org\" profile=\"monitor-all\" action=\"passthrough\" reqtype=\"direct\" url=\"https://test.org/\" sentbyte=111 rcvdbyte=1 direction=\"outgoing\" msg=\"tet msg\" method=\"domain\" cat=11 catdesc=\"test\""
}
{
          "tags" => [
        [0] "_grokparsefailure_sysloginput"
    ],
       "service" => {
        "type" => "system"
    },
      "@version" => "1",
    "@timestamp" => 2022-10-24T12:07:58.269123Z,
          "host" => {
        "ip" => "11.11.1.11"
    },
           "log" => {
        "syslog" => {
            "facility" => {
                "code" => 0,
                "name" => "kernel"
            },
            "priority" => 0,
            "severity" => {
                "code" => 0,
                "name" => "Emergency"
            }
        }
    },
         "event" => {
        "original" => nil
    },
       "message" => "<190>1 2022-10-24T11:07:57Z test - - - - eventtime=1111102477071520338 tz=\"+1111\" logid=\"1111128111\" type=\"utm\" subtype=\"test\" eventtype=\"signature\" level=\"information\" vd=\"root\" appid=11111 srcip=11.1.11.11 dstip=20.61.97.195 srcport=11110 dstport=111 srcintf=\"test1111\" srcintfrole=\"bvd\" dstintf=\"test\" dstintfrole=\"wan\" proto=1 service=\"SSL\" direction=\"outgoing\" policyid=11sessionid=111111111 applist=\"default\" action=\"pass\" appcat=\"test.Service\" app=\"SSL\" hostname=\"test.io\" incidentserialno=1111111 url=\"/\" msg=\"test: SSL,\" apprisk=\"elevated\" scertcname=\"*.test.io\" scertissuer=\"test Issuing CA 06\""
}
{
          "tags" => [
        [0] "_grokparsefailure_sysloginput"
    ],
       "service" => {
        "type" => "system"
    },
      "@version" => "1",
    "@timestamp" => 2022-10-24T11:07:58.269527Z,
          "host" => {
        "ip" => "11.11.1.11"
    },
           "log" => {
        "syslog" => {
            "facility" => {
                "code" => 0,
                "name" => "kernel"
            },
            "priority" => 0,
            "severity" => {
                "code" => 0,
                "name" => "Emergency"
            }
        }
    },
         "event" => {
        "original" => nil
    },
    ```

I'm not sure what is the issue, you are still using the syslog input?

You should use the tcp input to receive the fortinet logs and parse it with dissect and kv.

The syslog input is pretty limited, for example it does not support the rfc5425 you are using, it only supports syslog messages that follows RFC3164.

To parse the Fortinet logs you need to configure the format to default in the device and use the TCP input with the example pipeline I shared in a previous answer.

I was using the tcp input only {as assisted by you on Friday} along with the "default" syslog format; with that setup, I was not getting the Json output.
After changing the syslog format to rfc5425, I saw Json output over logstash TCP input {as mentioned in today's reply.
Regards,
Nivedita

Are you sure that your device is sending log using TCP and not UDP?

Check which protocol your device is sending log and configure it in the input.

The syslog input will not work in your case you need the tcp or udp input to receive the logs.

Try to open the same port both in TCP and UDP.

I already tested traffic using tcpdump -n port 8001 and I can see the events.
Even I installed the fortinet fortigate integration using elastic agent to check if that works, but no luck.

I'm not sure what is the issue if you are not receiving logs with the TCP input.

Try the following pipeline in Logstash:

input {
    tcp {
        port => 8001
    }
}
output {
    stdout {}
}

Check that your fortinet device is configured to send logs to your logstash server on port 8001 using TCP and the default format, if this is right you should receive the raw logs at least.

Thanks a lot @leandrojmp .
Someone has change the settings again and now I can see the data over in UDP from logstash pipeline (as mentioned below).

Also, now I am seeing events being inserted in elastic by kibana integration over udp, I will skip logstash pipeline {as I was working on prod env and cannot use it for testing now).

Thanks a lot for your support, I would really like to appreciate your efforts to assist me throughout, I owe you a good coffee :slight_smile:

 },
       "message" => "<189>1 2022-10-24T12:07:57Z Test - - - - eventtime=1666602477071090516 tz=\"+1111\" logid=\"0117011111\" type=\"utm\" subtype=\"webfilter\" eventtype=\"test_allow\" level=\"notice\" vd=\"root\" policyid=11 sessionid=111111111 srcip=11.1.11.111 srcport=11111 srcintf=\"test1111\" srcintfrole=\"ebd\" dstip=11.11.111.11 dstport=111 dstintf=\"test\" dstintfrole=\"wan\" proto=1 service=\"HTTPS\" hostname=\"test.org\" profile=\"monitor-all\" action=\"passthrough\" reqtype=\"direct\" url=\"https://test.org/\" sentbyte=111 rcvdbyte=1 direction=\"outgoing\" msg=\"tet msg\" method=\"domain\" cat=11 catdesc=\"test\""
}
{
    "@timestamp" => 2022-10-24T14:38:52.986895Z,
      "@version" => "1",
         "event" => {
        "original" => "<189>1 2022-10-24T12:07:57Z Test - - - - eventtime=1666602477071090516 tz=\"+1111\" logid=\"0117011111\" type=\"utm\" subtype=\"webfilter\" eventtype=\"test_allow\" level=\"notice\" vd=\"root\" policyid=11 sessionid=111111111 srcip=11.1.11.111 srcport=11111 srcintf=\"test1111\" srcintfrole=\"ebd\" dstip=11.11.111.11 dstport=111 dstintf=\"test\" dstintfrole=\"wan\" proto=1 service=\"HTTPS\" hostname=\"test.org\" profile=\"monitor-all\" action=\"passthrough\" reqtype=\"direct\" url=\"https://test.org/\" sentbyte=111 rcvdbyte=1 direction=\"outgoing\" msg=\"tet msg\" method=\"domain\" cat=11 catdesc=\"test\""
    }

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.