Logstash TCP Input Codecs

Elastic Stack Logstash
1

Is there a place to undestand exactly what format each of the TCP input codecs are meant to cover? (line vs json vs plain vs cef).
I have a situation of a new log source (Sophos firewall). Must use TLS so syslog input is out. Using TCP input with SSL/TLS. However, I have no idea what the format of of the incoming log is ahead of time. Sophos documentation just says syslog default. I've started to receive some logs and for now just ingesting them in logstash and dumping to a text file and what I see is that I will get like 100 events but its all in a single line, no line breaks. This is using codec plain.

Things I don't know:
Is there a better codec to choose? How would I troubleshoot? Just trial and error?
How about a website where you enter in the log and they tell you the codec to use?

2

If you are consuming events then you probably have the right codec on the input. Writing them to a file I suggest something like

file {
    path => "/some/where"
    codec => plain { format => "%{message}
" }
}

or

file {
    path => "/some/where"
    codec => line { format => "%{message}" }
}

"syslog" covers several different log formats so you would have to show us some example messages and we can help with a grok format.

1 Like
3

Like I said, the logs come in as one huge line. Here is a snippet, I dont even know where an individual log should start or end. I need help figuring out how to read this in so its individual lines per log.

nat_rule_id=\"13\" fw_rule_type=\"USER\" app_name=\"DNS\" app_risk=1 app_technology=\"Network Protocol\" app_category=\"Infrastructure\" ether_type=\"IPv4 (0x0800)\" in_interface=\"Port7\" src_mac=\"00:50:56:86:f2:23\" src_ip=\"10.0.228.35\" src_country=\"R1\" dst_ip=\"10.0.254.14\" dst_country=\"R1\" protocol=\"ICMP\" icmp_type=3 icmp_code=3 src_zone_type=\"LAN\" src_zone=\"LAN\" con_event=\"Interim\" con_id=\"699288\" hb_status=\"No Heartbeat\" app_resolved_by=\"Signature\" app_is_cloud=\"FALSE\" qualifier=\"New\" in_display_interface=\"10.0.228.1_22_Port7LAN\"\u0000<29>device_name=\"SFW\" timestamp=\"2024-01-07T01:12:59+0530\" device_model=\"XG310\" device_serial_id=\"C321014WTFF4\" log_id=\"018201500005\" log_type=\"Firewall\" log_component=\"ICMP ERROR MESSAGE\" log_subtype=\"Allowed\" log_version=1 severity=\"Notice\" fw_rule_id=\"24\" nat_rule_id=\"13\" fw_rule_type=\"USER\" app_name=\"DNS\" app_risk=1 app_technology=\"Network Protocol\" app_category=\"Infrastructure\" ether_type=\"IPv4 (0x0800)\" in_interface=\"Port1\" src_mac=\"e4:55:a8:1a:91:cb\" src_ip=\"10.0.253.221\" src_country=\"R1\" dst_ip=\"10.0.254.14\" dst_country=\"R1\" protocol=\"ICMP\" icmp_type=3 icmp_code=3 src_zone_type=\"LAN\" src_zone=\"LAN\" con_event=\"Interim\" con_id=\"3221815230\" hb_status=\"No Heartbeat\" app_resolved_by=\"Signature\" app_is_cloud=\"FALSE\" qualifier=\"New\" in_display_interface=\"10.0.253.1_24_Port1LAN\"\u0000<29>device_name=\"SFW\" timestamp=\"2024-01-07T01:12:59+0530\" device_model=\"XG310\" device_serial_id=\"C321014WTFF4\" log_id=\"018201500005\" log_type=\"Firewall\" log_component=\"ICMP ERROR MESSAGE\" log_subtype=\"Allowed\" log_version=1 severity=\"Notice\" fw_rule_id=\"26\" nat_rule_id=\"14\" fw_rule_type=\"USER\" app_name=\"LDAP Protocol\" app_risk=1 app_technology=\"Network Protocol\" app_category=\"Infrastructure\" ether_type=\"IPv4 (0x0800)\" in_interface=\"ipsec0\" src_mac=\"02:23:9c:f4:00:cc\" src_ip=\"64.147.195.210\" src_country=\"USA\" dst_ip=\"10.0.253.20\" dst_country=\"R1\" protocol=\"ICMP\" icmp_type=3 icmp_code=1 src_zone_type=\"VPN\" src_zone=\"VPN\" con_event=\"Interim\" con_id=\"1074141217\" hb_status=\"No Heartbeat\" app_resolved_by=\"Signature\" app_is_cloud=\"FALSE\" qualifier=\"New\" in_display_interface=\"ipsec0\"\u0000<29>device_name=\"SFW\" timestamp=\"2024-01-07T01:13:01+0530\" device_model=\"XG310\" device_serial_id=\"C321014WTFF4\" log_id=\"018201500005\" log_type=\"Firewall\" log_component=\"ICMP ERROR MESSAGE\" log_subtype=\"Allowed\" log_version=1 severity=\"Notice\" fw_rule_id=\"24\" nat_rule_id=\"13\" fw_rule_type=\"USER\" app_name=\"DNS\" app_risk=1 app_technology=\"Network Protocol\" app_category=\"Infrastructure\" ether_type=\"IPv4 (0x0800)\" in_interface=\"Port1\" src_mac=\"8c:ec:4b:d3:40:b9\" src_ip=\"10.0.253.246\" src_country=\"R1\" dst_ip=\"10.0.254.14\" dst_country=\"R1\" protocol=\"ICMP\" icmp_type=3 icmp_code=3 src_zone_type=\"LAN\" src_zone=\"LAN\" con_event=\"Interim\" con_id=\"3222142910\" hb_status=\"No Heartbeat\" app_resolved_by=\"Signature\" app_is_cloud=\"FALSE\" qualifier=\"New\" in_display_interface=\"10.0.253.1_24_Port1LAN\"\u0000<29>device_name=\"SFW\" timestamp=\"2024-01-07T01:13:01+0530\" device_model=\"XG310\" device_serial_id=\"C321014WTFF4\" log_id=\"018201500005\" log_type=\"Firewall\" log_component=\"ICMP ERROR MESSAGE\" log_subtype=\"Allowed\" log_version=1 severity=\"Notice\" fw_rule_id=\"24\" nat_rule_id=\"13\" fw_rule_type=\"USER\" app_name=\"DNS\" app_risk=1 app_technology=\"Network Protocol\" app_category=\"Infrastructure\" ether_type=\"IPv4 (0x0800)\" in_interface=\"Port1\" src_mac=\"e4:55:a8:1a:a8:b6\" src_ip=\"10.0.253.226\" src_country=\"R1\" dst_ip=\"10.0.254.14\" dst_country=\"R1\" protocol=\"ICMP\" icmp_type=3 icmp_code=3 src_zone_type=\"LAN\" src_zone=\"LAN\" con_event=\"Interim\" con_id=\"1074855870\" hb_status=\"No Heartbeat\" app_resolved_by=\"Signature\" app_is_cloud=\"FALSE\" qualifier=\"New\" in_display_interface=\"10.0.253.1_24_Port1LAN\"\u0000<29>device_name=\"SFW\" timestamp=\"2024-01-07T01:13:04+0530\" device_model=\"XG310\" device_serial_id=\"C321014WTFF4\" log_id=\"018201500005\" log_type=\"Firewall\" log_component=\"ICMP ERROR MESSAGE\

You can see the repetition in there to confirm these are multiple events all in one line. So far in my tcp input, I've tried codec plain and cef. No difference.
4

This looks like an issue on the Sophos side, the messages need to have a line break in the end of each event, which is not the case.

Logstash needs the line break on the end of the events.

While you may be able to split this message on multiple events using a split filter, I would recommend that you first try to see if you can fix this on Sophos side.

1 Like
5

That looks to be the line delimiter...

@leandrojmp, can he just set the delimiter on TCP input to that?

6

I don't think the logstash configuration language has any way to set the delimiter to that value.

1 Like
7

Not sure, it may work on some cases, but I'm not sure if this would work as setting the codec to line with this as the delimiter, never used like that.

Another approach would be to use a split filter and split the message on this value or maybe on the <29> value that comes in the beginning of every message.

But this is an issue on the sender side, it is not sending the messages correctly and even if you can split the messages in Logstash you will probably lose some messages that are being broken in multiple tcp messages, which seems to be the case of the message shared by the OP.

For example It starts with nat_rule_id, which is a field that comes further in the log message as you can see in the other messages in the same sample, it should start with <29> and the first field is always the device_name.

Unfortunately some vendors do a poor job with logs, I had a similar issue with some Cisco devices that was breaking up log messages into multiple tcp messages.

My suggestion is to try to see if this can be fixed on the sender side and if not work around with some logstash filters to get at least some of the messages.

1 Like
8

So I tried

input {
  tcp {
    port => 5044
    codec => line { delimiter => "\u0000" }
  }
}

Netcated that data to it

cat temp.log | nc -t localhost 5044

And it pulled out separate lines

{
      "@version" => "1",
       "message" => "<29>device_name=\\\"SFW\\\" timestamp=\\\"2024-01-07T01:12:59+0530\\\" device_model=\\\"XG310\\\" device_serial_id=\\\"C321014WTFF4\\\" log_id=\\\"018201500005\\\" log_type=\\\"Firewall\\\" log_component=\\\"ICMP ERROR MESSAGE\\\" log_subtype=\\\"Allowed\\\" log_version=1 severity=\\\"Notice\\\" fw_rule_id=\\\"26\\\" nat_rule_id=\\\"14\\\" fw_rule_type=\\\"USER\\\" app_name=\\\"LDAP Protocol\\\" app_risk=1 app_technology=\\\"Network Protocol\\\" app_category=\\\"Infrastructure\\\" ether_type=\\\"IPv4 (0x0800)\\\" in_interface=\\\"ipsec0\\\" src_mac=\\\"02:23:9c:f4:00:cc\\\" src_ip=\\\"64.147.195.210\\\" src_country=\\\"USA\\\" dst_ip=\\\"10.0.253.20\\\" dst_country=\\\"R1\\\" protocol=\\\"ICMP\\\" icmp_type=3 icmp_code=1 src_zone_type=\\\"VPN\\\" src_zone=\\\"VPN\\\" con_event=\\\"Interim\\\" con_id=\\\"1074141217\\\" hb_status=\\\"No Heartbeat\\\" app_resolved_by=\\\"Signature9999\\\" app_is_cloud=\\\"FALSE\\\" qualifier=\\\"New\\\" in_display_interface=\\\"ipsec0\\\"",
    "@timestamp" => 2024-01-07T02:01:54.768134Z,
         "event" => {
        "original" => "<29>device_name=\\\"SFW\\\" timestamp=\\\"2024-01-07T01:12:59+0530\\\" device_model=\\\"XG310\\\" device_serial_id=\\\"C321014WTFF4\\\" log_id=\\\"018201500005\\\" log_type=\\\"Firewall\\\" log_component=\\\"ICMP ERROR MESSAGE\\\" log_subtype=\\\"Allowed\\\" log_version=1 severity=\\\"Notice\\\" fw_rule_id=\\\"26\\\" nat_rule_id=\\\"14\\\" fw_rule_type=\\\"USER\\\" app_name=\\\"LDAP Protocol\\\" app_risk=1 app_technology=\\\"Network Protocol\\\" app_category=\\\"Infrastructure\\\" ether_type=\\\"IPv4 (0x0800)\\\" in_interface=\\\"ipsec0\\\" src_mac=\\\"02:23:9c:f4:00:cc\\\" src_ip=\\\"64.147.195.210\\\" src_country=\\\"USA\\\" dst_ip=\\\"10.0.253.20\\\" dst_country=\\\"R1\\\" protocol=\\\"ICMP\\\" icmp_type=3 icmp_code=1 src_zone_type=\\\"VPN\\\" src_zone=\\\"VPN\\\" con_event=\\\"Interim\\\" con_id=\\\"1074141217\\\" hb_status=\\\"No Heartbeat\\\" app_resolved_by=\\\"Signature9999\\\" app_is_cloud=\\\"FALSE\\\" qualifier=\\\"New\\\" in_display_interface=\\\"ipsec0\\\""
    }
}
{
      "@version" => "1",
       "message" => "<29>device_name=\\\"SFW\\\" timestamp=\\\"2024-01-07T01:12:59+0530\\\" device_model=\\\"XG310\\\" device_serial_id=\\\"C321014WTFF4\\\" log_id=\\\"018201500005\\\" log_type=\\\"Firewall\\\" log_component=\\\"ICMP ERROR MESSAGE\\\" log_subtype=\\\"Allowed\\\" log_version=1 severity=\\\"Notice\\\" fw_rule_id=\\\"24\\\" nat_rule_id=\\\"13\\\" fw_rule_type=\\\"USER\\\" app_name=\\\"DNS\\\" app_risk=1 app_technology=\\\"Network Protocol\\\" app_category=\\\"Infrastructure\\\" ether_type=\\\"IPv4 (0x0800)\\\" in_interface=\\\"Port1\\\" src_mac=\\\"e4:55:a8:1a:91:cb\\\" src_ip=\\\"10.0.253.221\\\" src_country=\\\"R1\\\" dst_ip=\\\"10.0.254.14\\\" dst_country=\\\"R1\\\" protocol=\\\"ICMP\\\" icmp_type=3 icmp_code=3 src_zone_type=\\\"LAN\\\" src_zone=\\\"LAN\\\" con_event=\\\"Interim\\\" con_id=\\\"3221815230\\\" hb_status=\\\"No Heartbeat\\\" app_resolved_by=\\\"Signature\\\" app_is_cloud=\\\"FALSE\\\" qualifier=\\\"New\\\" in_display_interface=\\\"10.0.253.1_24_Port1LAN\\\"",
    "@timestamp" => 2024-01-07T02:01:54.767765Z,
         "event" => {
        "original" => "<29>device_name=\\\"SFW\\\" timestamp=\\\"2024-01-07T01:12:59+0530\\\" device_model=\\\"XG310\\\" device_serial_id=\\\"C321014WTFF4\\\" log_id=\\\"018201500005\\\" log_type=\\\"Firewall\\\" log_component=\\\"ICMP ERROR MESSAGE\\\" log_subtype=\\\"Allowed\\\" log_version=1 severity=\\\"Notice\\\" fw_rule_id=\\\"24\\\" nat_rule_id=\\\"13\\\" fw_rule_type=\\\"USER\\\" app_name=\\\"DNS\\\" app_risk=1 app_technology=\\\"Network Protocol\\\" app_category=\\\"Infrastructure\\\" ether_type=\\\"IPv4 (0x0800)\\\" in_interface=\\\"Port1\\\" src_mac=\\\"e4:55:a8:1a:91:cb\\\" src_ip=\\\"10.0.253.221\\\" src_country=\\\"R1\\\" dst_ip=\\\"10.0.254.14\\\" dst_country=\\\"R1\\\" protocol=\\\"ICMP\\\" icmp_type=3 icmp_code=3 src_zone_type=\\\"LAN\\\" src_zone=\\\"LAN\\\" con_event=\\\"Interim\\\" con_id=\\\"3221815230\\\" hb_status=\\\"No Heartbeat\\\" app_resolved_by=\\\"Signature\\\" app_is_cloud=\\\"FALSE\\\" qualifier=\\\"New\\\" in_display_interface=\\\"10.0.253.1_24_Port1LAN\\\""
    }
}

Of course that delimiter is actually a unicode character correct so you might be able to set the charset and I think you might actually need to put in \\\u0000 or something like that. Perhaps what @badger means...

I'm not saying that will work .. but I'm kinda surprised this test worked...

I agree, @leandrojmp. I think this is a sender-side issue... but it might be worth a try...

2 Likes
9

So I ended up punting and just reverting to non-TLS, using a TCP listener on port 514 and everything works fine as I expected it to.

However, I can't possibly be the only one to experience this. It seems like this is a logstash bug because the alternative is that two different firewall vendors do something wrong. Hoping somebody knows something that can help.

The issue. Both Foritgate and Sophos firewalls work fine when setting syslog to a logstash TCP listener. However, when you enable TLS on that listener and configure the firewalls to use TLS, something changes. The logs no longer are seen as one log per line, it becomes a continuous stream/jumble. So, for example, if you simply read in and then output to a file, when TLS is enabled, the output is one long line, each new event added to it, as opposed to separate lines per event. The consequence of that is that logstash chokes on trying to process and parse those events. It seems unlikely that two different firewall vendors are doing something wrong and even so, how could one event verify or prove that?
So it seems the most likely scenario is some bug in logstash tcp with TLS enabled.

Any ideas?

10

After this context, it seems that this issue is related to the framing being used in your messages.

Fortigate when using TCP to send logs will use the reliable mode, which uses RFC6587 for the framing, and in this case the messages will indeed no have a line break in the end.

I would assume that Sophos behave in a similar way.

I'm not sure if you can change it to send logs using TLS, but use a different framing, you would need to check with each vendor.

The issue is that this is not a Logstash bug, Logstash seems to not support this kind of framing in the messages, so this would need to be implemented by Elastic.

There is already a request to implement this in the syslog input, but unfortunately since Logstash does not get much attention from Elastic, I would not expect this to be implemented quickly.

An alternative would be to use Filebeat in front of Logstash just to receive the logs since filebeat does support the RFC6587 framing.

11

Try use kv filter that is split key and value based on =

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.