# Logstash TCP Input Codecs **URL:** \ **Category:** Logstash\ **Created:** [January 6, 2024, 9:59pm UTC](https://discuss.elastic.co/t/logstash-tcp-input-codecs/350517 "2024-01-06T21:59:31Z")\ **Posts on this page:** 12\ **Page:** 1
**Author:** ![mgotechlock](https://avatars.discourse-cdn.com/v4/letter/m/dc4da7/32.png) [@mgotechlock](https://discuss.elastic.co/u/mgotechlock)\ **Post date:** [January 6, 2024, 9:59pm UTC](https://discuss.elastic.co/t/logstash-tcp-input-codecs/350517/1 "2024-01-06T21:59:31Z")
Is there a place to undestand exactly what format each of the TCP input codecs are meant to cover? (line vs json vs plain vs cef). I have a situation of a new log source (Sophos firewall). Must use TLS so syslog input is out. Using TCP input with SSL/TLS. However, I have no idea what the format of of the incoming log is ahead of time. Sophos documentation just says syslog default. I've started to receive some logs and for now just ingesting them in logstash and dumping to a text file and what I see is that I will get like 100 events but its all in a single line, no line breaks. This is using codec plain. Things I don't know: Is there a better codec to choose? How would I troubleshoot? Just trial and error? How about a website where you enter in the log and they tell you the codec to use? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [January 7, 2024, 1:10am UTC](https://discuss.elastic.co/t/logstash-tcp-input-codecs/350517/2 "2024-01-07T01:10:47Z")
If you are consuming events then you probably have the right codec on the input. Writing them to a file I suggest something like ``` file { path => "/some/where" codec => plain { format => "%{message} " } } ``` or ``` file { path => "/some/where" codec => line { format => "%{message}" } } ``` "syslog" covers several different log formats so you would have to show us some example messages and we can help with a grok format. ---
**Author:** ![mgotechlock](https://avatars.discourse-cdn.com/v4/letter/m/dc4da7/32.png) [@mgotechlock](https://discuss.elastic.co/u/mgotechlock)\ **Post date:** [January 7, 2024, 1:28am UTC](https://discuss.elastic.co/t/logstash-tcp-input-codecs/350517/3 "2024-01-07T01:28:56Z")
Like I said, the logs come in as one huge line. Here is a snippet, I dont even know where an individual log should start or end. I need help figuring out how to read this in so its individual lines per log. `nat_rule_id=\"13\" fw_rule_type=\"USER\" app_name=\"DNS\" app_risk=1 app_technology=\"Network Protocol\" app_category=\"Infrastructure\" ether_type=\"IPv4 (0x0800)\" in_interface=\"Port7\" src_mac=\"00:50:56:86:f2:23\" src_ip=\"10.0.228.35\" src_country=\"R1\" dst_ip=\"10.0.254.14\" dst_country=\"R1\" protocol=\"ICMP\" icmp_type=3 icmp_code=3 src_zone_type=\"LAN\" src_zone=\"LAN\" con_event=\"Interim\" con_id=\"699288\" hb_status=\"No Heartbeat\" app_resolved_by=\"Signature\" app_is_cloud=\"FALSE\" qualifier=\"New\" in_display_interface=\"10.0.228.1_22_Port7LAN\"\u0000<29>device_name=\"SFW\" timestamp=\"2024-01-07T01:12:59+0530\" device_model=\"XG310\" device_serial_id=\"C321014WTFF4\" log_id=\"018201500005\" log_type=\"Firewall\" log_component=\"ICMP ERROR MESSAGE\" log_subtype=\"Allowed\" log_version=1 severity=\"Notice\" fw_rule_id=\"24\" nat_rule_id=\"13\" fw_rule_type=\"USER\" app_name=\"DNS\" app_risk=1 app_technology=\"Network Protocol\" app_category=\"Infrastructure\" ether_type=\"IPv4 (0x0800)\" in_interface=\"Port1\" src_mac=\"e4:55:a8:1a:91:cb\" src_ip=\"10.0.253.221\" src_country=\"R1\" dst_ip=\"10.0.254.14\" dst_country=\"R1\" protocol=\"ICMP\" icmp_type=3 icmp_code=3 src_zone_type=\"LAN\" src_zone=\"LAN\" con_event=\"Interim\" con_id=\"3221815230\" hb_status=\"No Heartbeat\" app_resolved_by=\"Signature\" app_is_cloud=\"FALSE\" qualifier=\"New\" in_display_interface=\"10.0.253.1_24_Port1LAN\"\u0000<29>device_name=\"SFW\" timestamp=\"2024-01-07T01:12:59+0530\" device_model=\"XG310\" device_serial_id=\"C321014WTFF4\" log_id=\"018201500005\" log_type=\"Firewall\" log_component=\"ICMP ERROR MESSAGE\" log_subtype=\"Allowed\" log_version=1 severity=\"Notice\" fw_rule_id=\"26\" nat_rule_id=\"14\" fw_rule_type=\"USER\" app_name=\"LDAP Protocol\" app_risk=1 app_technology=\"Network Protocol\" app_category=\"Infrastructure\" ether_type=\"IPv4 (0x0800)\" in_interface=\"ipsec0\" src_mac=\"02:23:9c:f4:00:cc\" src_ip=\"64.147.195.210\" src_country=\"USA\" dst_ip=\"10.0.253.20\" dst_country=\"R1\" protocol=\"ICMP\" icmp_type=3 icmp_code=1 src_zone_type=\"VPN\" src_zone=\"VPN\" con_event=\"Interim\" con_id=\"1074141217\" hb_status=\"No Heartbeat\" app_resolved_by=\"Signature\" app_is_cloud=\"FALSE\" qualifier=\"New\" in_display_interface=\"ipsec0\"\u0000<29>device_name=\"SFW\" timestamp=\"2024-01-07T01:13:01+0530\" device_model=\"XG310\" device_serial_id=\"C321014WTFF4\" log_id=\"018201500005\" log_type=\"Firewall\" log_component=\"ICMP ERROR MESSAGE\" log_subtype=\"Allowed\" log_version=1 severity=\"Notice\" fw_rule_id=\"24\" nat_rule_id=\"13\" fw_rule_type=\"USER\" app_name=\"DNS\" app_risk=1 app_technology=\"Network Protocol\" app_category=\"Infrastructure\" ether_type=\"IPv4 (0x0800)\" in_interface=\"Port1\" src_mac=\"8c:ec:4b:d3:40:b9\" src_ip=\"10.0.253.246\" src_country=\"R1\" dst_ip=\"10.0.254.14\" dst_country=\"R1\" protocol=\"ICMP\" icmp_type=3 icmp_code=3 src_zone_type=\"LAN\" src_zone=\"LAN\" con_event=\"Interim\" con_id=\"3222142910\" hb_status=\"No Heartbeat\" app_resolved_by=\"Signature\" app_is_cloud=\"FALSE\" qualifier=\"New\" in_display_interface=\"10.0.253.1_24_Port1LAN\"\u0000<29>device_name=\"SFW\" timestamp=\"2024-01-07T01:13:01+0530\" device_model=\"XG310\" device_serial_id=\"C321014WTFF4\" log_id=\"018201500005\" log_type=\"Firewall\" log_component=\"ICMP ERROR MESSAGE\" log_subtype=\"Allowed\" log_version=1 severity=\"Notice\" fw_rule_id=\"24\" nat_rule_id=\"13\" fw_rule_type=\"USER\" app_name=\"DNS\" app_risk=1 app_technology=\"Network Protocol\" app_category=\"Infrastructure\" ether_type=\"IPv4 (0x0800)\" in_interface=\"Port1\" src_mac=\"e4:55:a8:1a:a8:b6\" src_ip=\"10.0.253.226\" src_country=\"R1\" dst_ip=\"10.0.254.14\" dst_country=\"R1\" protocol=\"ICMP\" icmp_type=3 icmp_code=3 src_zone_type=\"LAN\" src_zone=\"LAN\" con_event=\"Interim\" con_id=\"1074855870\" hb_status=\"No Heartbeat\" app_resolved_by=\"Signature\" app_is_cloud=\"FALSE\" qualifier=\"New\" in_display_interface=\"10.0.253.1_24_Port1LAN\"\u0000<29>device_name=\"SFW\" timestamp=\"2024-01-07T01:13:04+0530\" device_model=\"XG310\" device_serial_id=\"C321014WTFF4\" log_id=\"018201500005\" log_type=\"Firewall\" log_component=\"ICMP ERROR MESSAGE\` ```auto You can see the repetition in there to confirm these are multiple events all in one line. So far in my tcp input, I've tried codec plain and cef. No difference. ``` ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [January 7, 2024, 1:40am UTC](https://discuss.elastic.co/t/logstash-tcp-input-codecs/350517/4 "2024-01-07T01:40:37Z")
> [@mgotechlock](#): > > Like I said, the logs come in as one huge line. Here is a snippet, I dont even know where an individual log should start or end. I need help figuring out how to read this in so its individual lines per log. This looks like an issue on the Sophos side, the messages need to have a line break in the end of each event, which is not the case. Logstash needs the line break on the end of the events. While you may be able to split this message on multiple events using a `split` filter, I would recommend that you first try to see if you can fix this on Sophos side. ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [January 7, 2024, 1:41am UTC](https://discuss.elastic.co/t/logstash-tcp-input-codecs/350517/5 "2024-01-07T01:41:36Z")
> [@mgotechlock](#): > > \u0000 That looks to be the line delimiter... @leandrojmp, can he just set the [delimiter](https://www.elastic.co/guide/en/logstash/8.11/plugins-codecs-line.html#plugins-codecs-line-delimiter) on TCP input to that? ---
**Author:** ![Badger](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/badger/32/25190_2.png) [@Badger](https://discuss.elastic.co/u/Badger)\ **Post date:** [January 7, 2024, 1:46am UTC](https://discuss.elastic.co/t/logstash-tcp-input-codecs/350517/6 "2024-01-07T01:46:54Z")
I don't think the logstash configuration language has any way to set the delimiter to that value. ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [January 7, 2024, 1:56am UTC](https://discuss.elastic.co/t/logstash-tcp-input-codecs/350517/7 "2024-01-07T01:56:21Z")
> [@stephenb](#): > > @leandrojmp, can he just set the [delimiter](https://www.elastic.co/guide/en/logstash/8.11/plugins-codecs-line.html#plugins-codecs-line-delimiter) on TCP input to that? Not sure, it may work on some cases, but I'm not sure if this would work as setting the `codec` to `line` with this as the delimiter, never used like that. Another approach would be to use a `split` filter and split the message on this value or maybe on the `<29>` value that comes in the beginning of every message. But this is an issue on the sender side, it is not sending the messages correctly and even if you can split the messages in Logstash you will probably lose some messages that are being broken in multiple tcp messages, which seems to be the case of the message shared by the OP. For example It starts with `nat_rule_id`, which is a field that comes further in the log message as you can see in the other messages in the same sample, it should start with `<29>` and the first field is always the `device_name`. Unfortunately some vendors do a poor job with logs, I had a similar issue with some Cisco devices that was breaking up log messages into multiple tcp messages. My suggestion is to try to see if this can be fixed on the sender side and if not work around with some logstash filters to get at least some of the messages. ---
**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\ **Post date:** [January 7, 2024, 2:09am UTC](https://discuss.elastic.co/t/logstash-tcp-input-codecs/350517/8 "2024-01-07T02:09:17Z")
So I tried ```auto input { tcp { port => 5044 codec => line { delimiter => "\u0000" } } } ``` Netcated that data to it `cat temp.log | nc -t localhost 5044` And it pulled out separate lines ```auto { "@version" => "1", "message" => "<29>device_name=\\\"SFW\\\" timestamp=\\\"2024-01-07T01:12:59+0530\\\" device_model=\\\"XG310\\\" device_serial_id=\\\"C321014WTFF4\\\" log_id=\\\"018201500005\\\" log_type=\\\"Firewall\\\" log_component=\\\"ICMP ERROR MESSAGE\\\" log_subtype=\\\"Allowed\\\" log_version=1 severity=\\\"Notice\\\" fw_rule_id=\\\"26\\\" nat_rule_id=\\\"14\\\" fw_rule_type=\\\"USER\\\" app_name=\\\"LDAP Protocol\\\" app_risk=1 app_technology=\\\"Network Protocol\\\" app_category=\\\"Infrastructure\\\" ether_type=\\\"IPv4 (0x0800)\\\" in_interface=\\\"ipsec0\\\" src_mac=\\\"02:23:9c:f4:00:cc\\\" src_ip=\\\"64.147.195.210\\\" src_country=\\\"USA\\\" dst_ip=\\\"10.0.253.20\\\" dst_country=\\\"R1\\\" protocol=\\\"ICMP\\\" icmp_type=3 icmp_code=1 src_zone_type=\\\"VPN\\\" src_zone=\\\"VPN\\\" con_event=\\\"Interim\\\" con_id=\\\"1074141217\\\" hb_status=\\\"No Heartbeat\\\" app_resolved_by=\\\"Signature9999\\\" app_is_cloud=\\\"FALSE\\\" qualifier=\\\"New\\\" in_display_interface=\\\"ipsec0\\\"", "@timestamp" => 2024-01-07T02:01:54.768134Z, "event" => { "original" => "<29>device_name=\\\"SFW\\\" timestamp=\\\"2024-01-07T01:12:59+0530\\\" device_model=\\\"XG310\\\" device_serial_id=\\\"C321014WTFF4\\\" log_id=\\\"018201500005\\\" log_type=\\\"Firewall\\\" log_component=\\\"ICMP ERROR MESSAGE\\\" log_subtype=\\\"Allowed\\\" log_version=1 severity=\\\"Notice\\\" fw_rule_id=\\\"26\\\" nat_rule_id=\\\"14\\\" fw_rule_type=\\\"USER\\\" app_name=\\\"LDAP Protocol\\\" app_risk=1 app_technology=\\\"Network Protocol\\\" app_category=\\\"Infrastructure\\\" ether_type=\\\"IPv4 (0x0800)\\\" in_interface=\\\"ipsec0\\\" src_mac=\\\"02:23:9c:f4:00:cc\\\" src_ip=\\\"64.147.195.210\\\" src_country=\\\"USA\\\" dst_ip=\\\"10.0.253.20\\\" dst_country=\\\"R1\\\" protocol=\\\"ICMP\\\" icmp_type=3 icmp_code=1 src_zone_type=\\\"VPN\\\" src_zone=\\\"VPN\\\" con_event=\\\"Interim\\\" con_id=\\\"1074141217\\\" hb_status=\\\"No Heartbeat\\\" app_resolved_by=\\\"Signature9999\\\" app_is_cloud=\\\"FALSE\\\" qualifier=\\\"New\\\" in_display_interface=\\\"ipsec0\\\"" } } { "@version" => "1", "message" => "<29>device_name=\\\"SFW\\\" timestamp=\\\"2024-01-07T01:12:59+0530\\\" device_model=\\\"XG310\\\" device_serial_id=\\\"C321014WTFF4\\\" log_id=\\\"018201500005\\\" log_type=\\\"Firewall\\\" log_component=\\\"ICMP ERROR MESSAGE\\\" log_subtype=\\\"Allowed\\\" log_version=1 severity=\\\"Notice\\\" fw_rule_id=\\\"24\\\" nat_rule_id=\\\"13\\\" fw_rule_type=\\\"USER\\\" app_name=\\\"DNS\\\" app_risk=1 app_technology=\\\"Network Protocol\\\" app_category=\\\"Infrastructure\\\" ether_type=\\\"IPv4 (0x0800)\\\" in_interface=\\\"Port1\\\" src_mac=\\\"e4:55:a8:1a:91:cb\\\" src_ip=\\\"10.0.253.221\\\" src_country=\\\"R1\\\" dst_ip=\\\"10.0.254.14\\\" dst_country=\\\"R1\\\" protocol=\\\"ICMP\\\" icmp_type=3 icmp_code=3 src_zone_type=\\\"LAN\\\" src_zone=\\\"LAN\\\" con_event=\\\"Interim\\\" con_id=\\\"3221815230\\\" hb_status=\\\"No Heartbeat\\\" app_resolved_by=\\\"Signature\\\" app_is_cloud=\\\"FALSE\\\" qualifier=\\\"New\\\" in_display_interface=\\\"10.0.253.1_24_Port1LAN\\\"", "@timestamp" => 2024-01-07T02:01:54.767765Z, "event" => { "original" => "<29>device_name=\\\"SFW\\\" timestamp=\\\"2024-01-07T01:12:59+0530\\\" device_model=\\\"XG310\\\" device_serial_id=\\\"C321014WTFF4\\\" log_id=\\\"018201500005\\\" log_type=\\\"Firewall\\\" log_component=\\\"ICMP ERROR MESSAGE\\\" log_subtype=\\\"Allowed\\\" log_version=1 severity=\\\"Notice\\\" fw_rule_id=\\\"24\\\" nat_rule_id=\\\"13\\\" fw_rule_type=\\\"USER\\\" app_name=\\\"DNS\\\" app_risk=1 app_technology=\\\"Network Protocol\\\" app_category=\\\"Infrastructure\\\" ether_type=\\\"IPv4 (0x0800)\\\" in_interface=\\\"Port1\\\" src_mac=\\\"e4:55:a8:1a:91:cb\\\" src_ip=\\\"10.0.253.221\\\" src_country=\\\"R1\\\" dst_ip=\\\"10.0.254.14\\\" dst_country=\\\"R1\\\" protocol=\\\"ICMP\\\" icmp_type=3 icmp_code=3 src_zone_type=\\\"LAN\\\" src_zone=\\\"LAN\\\" con_event=\\\"Interim\\\" con_id=\\\"3221815230\\\" hb_status=\\\"No Heartbeat\\\" app_resolved_by=\\\"Signature\\\" app_is_cloud=\\\"FALSE\\\" qualifier=\\\"New\\\" in_display_interface=\\\"10.0.253.1_24_Port1LAN\\\"" } } ``` Of course that delimiter is actually a unicode character correct so you might be able to set the charset and I think you might actually need to put in `\\\u0000` or something like that. Perhaps what @badger means... I'm not saying that will work .. but I'm kinda surprised this test worked... I agree, @leandrojmp. I think this is a sender-side issue... but it might be worth a try... ---
**Author:** ![mgotechlock](https://avatars.discourse-cdn.com/v4/letter/m/dc4da7/32.png) [@mgotechlock](https://discuss.elastic.co/u/mgotechlock)\ **Post date:** [January 8, 2024, 1:00pm UTC](https://discuss.elastic.co/t/logstash-tcp-input-codecs/350517/9 "2024-01-08T13:00:05Z")
So I ended up punting and just reverting to non-TLS, using a TCP listener on port 514 and everything works fine as I expected it to. However, I can't possibly be the only one to experience this. It seems like this is a logstash bug because the alternative is that two different firewall vendors do something wrong. Hoping somebody knows something that can help. The issue. Both Foritgate and Sophos firewalls work fine when setting syslog to a logstash TCP listener. However, when you enable TLS on that listener and configure the firewalls to use TLS, something changes. The logs no longer are seen as one log per line, it becomes a continuous stream/jumble. So, for example, if you simply read in and then output to a file, when TLS is enabled, the output is one long line, each new event added to it, as opposed to separate lines per event. The consequence of that is that logstash chokes on trying to process and parse those events. It seems unlikely that two different firewall vendors are doing something wrong and even so, how could one event verify or prove that? So it seems the most likely scenario is some bug in logstash tcp with TLS enabled. Any ideas? ---
**Author:** ![leandrojmp](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/leandrojmp/32/107231_2.png) [@leandrojmp](https://discuss.elastic.co/u/leandrojmp)\ **Post date:** [January 8, 2024, 2:00pm UTC](https://discuss.elastic.co/t/logstash-tcp-input-codecs/350517/10 "2024-01-08T14:00:06Z")
After this context, it seems that this issue is related to the framing being used in your messages. Fortigate when using TCP to send logs will use the `reliable` mode, which uses `RFC6587` for the framing, and in this case the messages will indeed no have a line break in the end. I would assume that Sophos behave in a similar way. I'm not sure if you can change it to send logs using TLS, but use a different framing, you would need to check with each vendor. The issue is that this is not a Logstash bug, Logstash seems to not support this kind of framing in the messages, so this would need to be implemented by Elastic. There is already a [request](https://github.com/logstash-plugins/logstash-input-syslog/issues/65) to implement this in the `syslog` input, but unfortunately since Logstash does not get much attention from Elastic, I would not expect this to be implemented quickly. An alternative would be to use Filebeat in front of Logstash just to receive the logs since filebeat does support the RFC6587 [framing](https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-tcp.html#filebeat-input-tcp-tcp-framing). ---
**Author:** ![e997cd7e8d9915436150](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/e997cd7e8d9915436150/32/40255_2.png) [@e997cd7e8d9915436150](https://discuss.elastic.co/u/e997cd7e8d9915436150)\ **Post date:** [January 11, 2024, 11:09pm UTC](https://discuss.elastic.co/t/logstash-tcp-input-codecs/350517/11 "2024-01-11T23:09:15Z")
> [@mgotechlock](#): > > So I ended up punting and just reverting to non-TLS, using a TCP listener on port 514 and everything works fine as I expected it to. > > However, I can't possibly be the only one to experience this. It seems like this is a logstash bug because the alternative is that two different firewall vendors do something wrong. Hoping somebody knows something that can help. > > The issue. Both Foritgate and Sophos firewalls work fine when setting syslog to a logstash TCP listener. However, when you enable TLS on that listener and configure the firewalls to use TLS, something changes. The logs no longer are seen as one log per line, it becomes a continuous stream/jumble. So, for example, if you simply read in and then output to a file, when TLS is enabled, the output is one long line, each new event added to it, as opposed to separate lines per event. The consequence of that is that logstash chokes on trying to process and parse those events. It seems unlikely that two different firewall vendors are doing something wrong and even so, how could one event verify or prove that? > So it seems the most likely scenario is some bug in logstash tcp with TLS enabled. Try use kv filter that is split key and value based on `=` ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [February 8, 2024, 11:09pm UTC](https://discuss.elastic.co/t/logstash-tcp-input-codecs/350517/12 "2024-02-08T23:09:55Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.