# FortiMail logs are being combined in TCP input **URL:** \ **Category:** Logstash\ **Created:** [April 11, 2023, 5:51pm UTC](https://discuss.elastic.co/t/fortimail-logs-are-being-combined-in-tcp-input/329768 "2023-04-11T17:51:47Z")\ **Posts on this page:** 6\ **Page:** 1
**Author:** ![6igwig](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/6igwig/32/88971_2.png) [@6igwig](https://discuss.elastic.co/u/6igwig)\ **Post date:** [April 11, 2023, 5:51pm UTC](https://discuss.elastic.co/t/fortimail-logs-are-being-combined-in-tcp-input/329768/1 "2023-04-11T17:51:47Z")
I have configured a tcp input in logstash to receive FortiMail logs. I believe the logs are losing their new line character in transit because all of the logs come in as a single document. (If I leave the pipeline running, nothing goes through it. Then when I restart logstash a humongous document is ingested all of the logs combined into the message field.) The only thing I've tried that has successfully broken the logs out into separate documents is adding the CSV codec to the TCP input. The CSV codec breaks the logs up and sends them to elastic, but I am getting this error message in the logstash pipeline: `CSV parse failure. Falling back to plain-text {:exception=>CSV::MalformedCSVError, :message=>"Illegal quoting in line 1.", :data=>"437 <6>date=2023-04-10,time=16:52:07.587,device_id=xxx,log_id=123,type=event,subtype=smtp,pri=information, user=mail,ui=mail,action=NONE,status=N/A,session_id=\"33AKq7LU018089-33AKq7LX018089\",msg=\"from=, size=8638, class=0, nrcpts=1, msgid=, bodytype=7BIT, proto=ESMTP, daemon=SMTP_MTA, relay=mail-as34dfasdf.outbound.protection.outlook.com [123.123.123.123]\""} ` I tried changing the codec to `plain`, but logstash reverts it automatically back to "line" (`Automatically switching from plain to line codec {:plugin=>"tcp"}`) which combines all logs into one. Any help would be greatly appreciated! ---
**Author:** ![6igwig](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/6igwig/32/88971_2.png) [@6igwig](https://discuss.elastic.co/u/6igwig)\ **Post date:** [April 11, 2023, 6:03pm UTC](https://discuss.elastic.co/t/fortimail-logs-are-being-combined-in-tcp-input/329768/2 "2023-04-11T18:03:04Z")
Some more info... The reason I am using the TCP input is because I want to be able to encrypt the logs in transit with the SSL settings. Here's what I have already tried.. - Replace CSV codec with CSV filter. This does not break the logs out. - Replace CSV codec with `syslog_pri` and `kv` filters - Played around with CSV codec parameters: `include_headers => false`, `columns => ['priority','log_data']` - Replaced TCP input with beats input ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [April 11, 2023, 6:16pm UTC](https://discuss.elastic.co/t/fortimail-logs-are-being-combined-in-tcp-input/329768/3 "2023-04-11T18:16:18Z")
Please share your logstash configuration. ---
**Author:** ![6igwig](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/6igwig/32/88971_2.png) [@6igwig](https://discuss.elastic.co/u/6igwig)\ **Post date:** [April 11, 2023, 7:10pm UTC](https://discuss.elastic.co/t/fortimail-logs-are-being-combined-in-tcp-input/329768/4 "2023-04-11T19:10:48Z")
```auto input { tcp { port => 1111 ssl_enable => true ssl_verify => false ssl_cert => '/etc/logstash/certs/my_certificate.crt' ssl_key => '/etc/logstash/certs/my_certificate.key' } } output { lumberjack { hosts => ["logstash.my_network.com"] port => 1111 ssl_certificate => "/etc/logstash/my_certificate.cert" codec => json } } ``` ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [April 11, 2023, 7:12pm UTC](https://discuss.elastic.co/t/fortimail-logs-are-being-combined-in-tcp-input/329768/5 "2023-04-11T19:12:53Z")
I suspect that the problem is FortiMail is sending RFC 5425 compliant "octet-counted" messages (i.e. no trailing newline). See [this](https://discuss.elastic.co/t/logstash-compliance-with-rfc5425-and-rfc5426/327243/3) thread for a little more detail. Not sure if FortiMail can be configured to use traditional format instead of RFC 5425. The existing tcp input is line-oriented, but you need a byte-oriented input. The difference between the two creates issues that were [long ago recognized](https://github.com/elastic/logstash/issues/5124). (Note that the issue is still open, none of the re-architecture that was envisioned actually happened.) You could re-write the tcp input to be byte oriented, but it would be much simpler to add a syslog server that can read RFC 5425 compliant messages from FortiMail and forward them to logstash in traditional newline separated format. > [@6igwig](#): > > ``` > :message=>"Illegal quoting in line 1.", :data=>"437 <6>date=2023-04-10,time=16:52:07.587,device_id=xxx,log_id=123,type=event,subtype=smtp,pri=information, user=mail,ui=mail,action=NONE,status=N/A,session_id=\"33AKq7LU018089-33AKq7LX018089\", > > ``` Your problem here is that if a CSV field contains double quotes then the entire field must be quoted. So instead of ``` session_id="33AKq7LU018089-33AKq7LX018089" ``` it would have to be ``` "session_id=""33AKq7LU018089-33AKq7LX018089""" ``` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [May 9, 2023, 7:13pm UTC](https://discuss.elastic.co/t/fortimail-logs-are-being-combined-in-tcp-input/329768/6 "2023-05-09T19:13:35Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.