FortiMail logs are being combined in TCP input

[6igwig (James)]

I have configured a tcp input in logstash to receive FortiMail logs. I believe the logs are losing their new line character in transit because all of the logs come in as a single document. (If I leave the pipeline running, nothing goes through it. Then when I restart logstash a humongous document is ingested all of the logs combined into the message field.)

The only thing I've tried that has successfully broken the logs out into separate documents is adding the CSV codec to the TCP input. The CSV codec breaks the logs up and sends them to elastic, but I am getting this error message in the logstash pipeline: CSV parse failure. Falling back to plain-text {:exception=>CSV::MalformedCSVError, :message=>"Illegal quoting in line 1.", :data=>"437 <6>date=2023-04-10,time=16:52:07.587,device_id=xxx,log_id=123,type=event,subtype=smtp,pri=information, user=mail,ui=mail,action=NONE,status=N/A,session_id=\"33AKq7LU018089-33AKq7LX018089\",msg=\"from=<xxx@xxx.com>, size=8638, class=0, nrcpts=1, msgid=<asdfasd@asdfds>, bodytype=7BIT, proto=ESMTP, daemon=SMTP_MTA, relay=mail-as34dfasdf.outbound.protection.outlook.com [123.123.123.123]\""}

I tried changing the codec to plain, but logstash reverts it automatically back to "line" (Automatically switching from plain to line codec {:plugin=>"tcp"}) which combines all logs into one.

Any help would be greatly appreciated!

[6igwig (James)]

Some more info... The reason I am using the TCP input is because I want to be able to encrypt the logs in transit with the SSL settings.

Here's what I have already tried..

• Replace CSV codec with CSV filter. This does not break the logs out.
• Replace CSV codec with syslog_pri and kv filters
• Played around with CSV codec parameters: include_headers => false, columns => ['priority','log_data']
• Replaced TCP input with beats input

[leandrojmp (Leandro Pereira)]

Please share your logstash configuration.

[6igwig (James)]

input {
    tcp {
        port => 1111
        ssl_enable => true
        ssl_verify => false
        ssl_cert => '/etc/logstash/certs/my_certificate.crt'
        ssl_key => '/etc/logstash/certs/my_certificate.key'
    }
}
output {
    lumberjack {
        hosts => ["logstash.my_network.com"]
        port => 1111
        ssl_certificate => "/etc/logstash/my_certificate.cert"
        codec => json
    }
}

[Badger]

I suspect that the problem is FortiMail is sending RFC 5425 compliant "octet-counted" messages (i.e. no trailing newline). See this thread for a little more detail. Not sure if FortiMail can be configured to use traditional format instead of RFC 5425.

The existing tcp input is line-oriented, but you need a byte-oriented input. The difference between the two creates issues that were long ago recognized. (Note that the issue is still open, none of the re-architecture that was envisioned actually happened.)

You could re-write the tcp input to be byte oriented, but it would be much simpler to add a syslog server that can read RFC 5425 compliant messages from FortiMail and forward them to logstash in traditional newline separated format.

@6igwig

:message=>"Illegal quoting in line 1.", :data=>"437 <6>date=2023-04-10,time=16:52:07.587,device_id=xxx,log_id=123,type=event,subtype=smtp,pri=information, user=mail,ui=mail,action=NONE,status=N/A,session_id=\"33AKq7LU018089-33AKq7LX018089\",

Your problem here is that if a CSV field contains double quotes then the entire field must be quoted. So instead of

session_id="33AKq7LU018089-33AKq7LX018089"

it would have to be

"session_id=""33AKq7LU018089-33AKq7LX018089"""