Fields not working in Syslog output

Elastic Stack Logstash
1

I'm attempting to convert netflow data from the netflow codec into syslog data using the syslog output plugin.

The issue i'm having is that no matter what I set the syslog plugin "message" to, it only ever outputs the default of "%{message}". I have tried changing it to anything and everything and the setting in the config seems to have absolutely no affect on the output.

Here's my config:

input {
  udp {
    port  => 2055
    codec => netflow
  }
}
output {
  syslog{
    host => "127.0.0.1"
    port => 514
    protocol => "tcp"
    facility => "local0"
    severity => "informational"
    appname => "netflow"
    sourcehost => "host"
    message => "%{netflow}"
    }
}

Here is the stdout from the input:

{
       "netflow" => {
              "flow_seq_num" => 3,
             "ipv4_dst_addr" => "77.12.190.94",
               "engine_type" => 1,
                    "dst_as" => 13887,
             "last_switched" => "2019-12-10T23:12:03.969Z",
                  "in_bytes" => 921,
               "l4_src_port" => 9010,
                    "src_as" => 60550,
                  "dst_mask" => 26,
                   "src_tos" => 0,
                   "version" => 5,
             "ipv4_src_addr" => "10.154.20.12",
         "sampling_interval" => 0,
               "output_snmp" => 0,
            "first_switched" => "2019-12-10T23:12:03.540Z",
                  "protocol" => 6,
                 "tcp_flags" => 0,
        "sampling_algorithm" => 0,
             "ipv4_next_hop" => "150.20.145.1",
                 "engine_id" => 0,
              "flow_records" => 8,
                "input_snmp" => 0,
                   "in_pkts" => 496,
               "l4_dst_port" => 3306,
                  "src_mask" => 0
    },
    "@timestamp" => 2019-12-10T23:12:03.181Z,
          "host" => "172.17.0.2",
      "@version" => "1"
}

Here's the output, note that no matter what i put in the config, the message output is always "%{message}". What am i doing wrong?

Dec 10 23:03:07 host netflow[-]: 2019-12-10T23:03:07.833Z 172.17.0.2 %{message}

2

If you have set 'message => "%{netflow}"' and you end up with the default "%{message}" then that suggests you are not running with the configuration you think you are.

Try running with --config.debug --log.level debug --config.test_and_exit on the command line. That will show you each file that it is loading as part of the configuration, and it will show you the merged configuration.

3

Here's the result, from what I'm reading everything is correct. I must be doing something wrong though. In this case I told message to just be the word "front" and it still output "%{message}". Thanks for your help @Badger

https://pastebin.com/SJhkjver

4

So to get around this bug? I added:

filter {
        mutate {
                add_field => {"message" => "test"}
        }
}

And now the syslog message option outputs whatever I put in that field i created. The problem is I still can't seem to access ANY of the values I input from netflow.

I tried doing add_field => {"message" => "%{netflow}"} and it just prints "${netflow}"
I tried doing add_field => {"message" => "[netflow]"} and it just prints "[netflow]"
I tried doing add_field => {"message" => "[netflow][in_bytes]"} and you guessed it, it prints "[netflow][in_bytes]

How do I access the values input from this:

{
       "netflow" => {
              "flow_seq_num" => 3,
             "ipv4_dst_addr" => "77.12.190.94",
               "engine_type" => 1,
                    "dst_as" => 13887,
             "last_switched" => "2019-12-10T23:12:03.969Z",
                  "in_bytes" => 921,
               "l4_src_port" => 9010,
                    "src_as" => 60550,
                  "dst_mask" => 26,
                   "src_tos" => 0,
                   "version" => 5,
             "ipv4_src_addr" => "10.154.20.12",
         "sampling_interval" => 0,
               "output_snmp" => 0,
            "first_switched" => "2019-12-10T23:12:03.540Z",
                  "protocol" => 6,
                 "tcp_flags" => 0,
        "sampling_algorithm" => 0,
             "ipv4_next_hop" => "150.20.145.1",
                 "engine_id" => 0,
              "flow_records" => 8,
                "input_snmp" => 0,
                   "in_pkts" => 496,
               "l4_dst_port" => 3306,
                  "src_mask" => 0
    },
    "@timestamp" => 2019-12-10T23:12:03.181Z,
          "host" => "172.17.0.2",
      "@version" => "1"
}
5

It seems there is a bug in both the netflow codec and the syslog plugin that created the perfect storm.

the default variable for netflow is "netflow" but it doesn't work. You have to manually define the variable

codec => netflow { target => variable_name}

Then you can use

%{variable_name}

to access the data

Then with syslog, the default message is %{message} and there's no way to change it. So I had to mutate:

add_field {"message" => "%{variable_name}}

6

I had the same issue with the syslog output plugin, I'm pretty sure that the variable message of the pipeline leaks into the plugins message variable, so you get a literal %{message}. So I changed the plugins variable to raw_message and my problem was solved. I also had issues with the TLS retry handling so I forked the repo and now pushed my changes. I haven't contacted the logstash-plugin maintainers or created a pull request, but I don't see much action in their repo on other pull requests.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.