How do you tag by multiple source IPs?

#1

Hi,

I'm transitioning to logstash from an rsyslog config that ingests logs from a list of network devices (routers, switches, firewalls), then does something based on the source IP. Here's an abbreviated example:

if ($fromhost-ip == '10.250.0.73' or $fromhost-ip == '10.253.1.161' ) then @10.250.0.208

The way I was thinking about implementing this in logstash is adding a tag if the source IP matched. Would this be a similar config?

filter {
  if [source] == '10.250.0.73' or [source] == '10.253.1.161' {
    mutate {
      add_tag => "network_device"
    }
  }
}

output {
  if "network_device" in [tags] {
    udp {
      host => "10.250.0.208"
      port => 514
    }
  }
}

Or is there a better way to do this? Can you do a regex =~ "(10.250.0.73|10.253.1.161)" or anything similar?

I'm testing looking for that specific tag and writing it out to a file but then also have a catch-all if nothing else matches and it's hitting that catch-all so I'm apparently not doing something right.

There are a lot of other log source types coming in so I didn't know if applying tags determined by source IP, message string, etc, is the best methodology (sorry, I'm just getting started on learning this) but I'd love to learn some tips if possible.

Thanks!

#2

You can do that, or you can do the regex that you showed, or you can use a translate filter.

#3

Thank you for your reply. I'm still not quite sure what's wrong with my configuration when I'm trying to add a tag based on source IP:

root@localhost [/etc/logstash/conf.d]
# cat 02-test-filter.conf 
filter {
  if [source] == "10.190.200.30" {
    mutate { 
      add_tag => "Test" }
  }
}

Then I write it to test.log if it has the "Test" tag:

root@localhost [/etc/logstash/conf.d]
# cat 03-test-output.conf 
output {
  if "Test" in [tags] {
    file {
      codec => "line"
      path =>  "/syslog/test.log" }
  }
}

But it's hitting my catchall rule even though the source IP is what I defined in the above filter:

root@localhost [/syslog]
# cat unmatched.log 
2019-10-10T18:03:57.186Z 10.190.200.30 Test

I'm sure I'm just doing something stupid but I can't figure it out. Can anyone point me in the right direction?

Thanks again!

#4

If you point path.config to a directory then all of the files in the directory (every file) is concatenated to create the configuration. Events are read from each input, run through all of the filters, then sent to all of the outputs (unless you use conditionals). Does that match your understanding?

#5

Yes, sir. I've ripped everything else out so there are only the inputs, the one filter, and the two output rules:

root@localhost [/etc/logstash/conf.d]
# cat 02-test-filter.conf 
filter {
  if [source] == "10.190.200.30" {
    mutate { 
      add_tag => "Test" }
  }
}

Thu 10 Oct 19:24:19
root@localhost [/etc/logstash/conf.d]
# cat 03-file-output.conf 
output {

  if "Test" in [tags] {
    file {
      codec => "line"
      path =>  "/syslog/test.log" }
  }

  else {
    file {
      codec => "line"
      path => "/syslog/unmatched.log" }
  }

}

But it's not catching that 'Test' tag (or maybe I'm doing it the wrong way?). All of the other rules I'd created that are based on strings in the message or tags set by Beats are working fine. It's just this one based on the source IP that I can't quite get.

Thank you again for your reply -- I really appreciate it!

#6

Please change the codec to rubydebug and post what the [source] field looks like. Maybe it is an array or something.

#7

Here's what I get:

{
          "host" => "10.190.200.30",
    "@timestamp" => 2019-10-10T20:38:22.310Z,
       "message" => "Test",
      "@version" => "1"
}

So do I just need to change it to be 'if [host] == "10.190.200.30"'... rather than [source]?

Thanks again!

#8

Sorry, I should have tested it before asking. But it does look like that's the case. Not sure if that's the new syntax for 7.4, but [host] does work and [source] does not:

{
    "@timestamp" => 2019-10-10T20:50:33.916Z,
       "message" => "Test",
          "host" => "10.190.200.30",
      "@version" => "1",
          "tags" => [
        [0] "Test"
    ]
}

Thank you so much for your help!