Using cidr plugin for tagging logs

Elastic Stack Logstash
1

Hi everyone,

Before I start I have to warn you that i'm French and my English is kind limited.

I just started with ELK but I want to so something like this :

host A --> logs --
LANA             |
                   --> rsyslog server --> input logstash syslog --> filter cidr ( tag according to the source network of the log ) --> output : if tag lanA, put it in index LANA, else if tag lanB, put it in index LANB, else stdout --> elasticsearch
host B --> logs--|
LANB

the cidr filter for logstash should check the source network, and tag the log with a specific term according to the source log but it's not working ... Is someone already try this filter with this function ?

my config file look like to that :

 filter {
	cidr {	
		network => "172.16.0.0/16"
		add_tag => ["lan1"]
	}


        cidr {
    	        network => "192.168.0.0/24"
		add_tag => ["lan2"]
	}
}

output {
	if "lan1" in [tags] {
		elasticsearch {
			hosts => "localhost:9200"
			index => "lan1"
		}
	}
	else if "lan2" in [tags] {
		elasticsearch {
			hosts => "localhost:9200"
			index => "lan2"
		}
	}	

else {
	stdout {
		codec => rubydebug	
	}
}

But all logs which provides from hosts on network 192.168.0.0/24 and 172.16.0.0/16 are displayed on stdout.
Did I make mistakes on the syntax ? Is someone already use this plugin to do that ?

Thanks a lot for your help.

2

You don't need the cidr plugin for that. A simple regex can do it.

If network =~ 192.168.0.([0-9]{0,3}) {

I'm doing this from my phone so you'll need to check the syntax but that will work.

I do that for all my locations.

If you need help, let me know. I'll grab a copy of a config when I get a chance and post it.

3

The OP's problem is that the address option must be set to one or more addresses to check. Right now the filter is configured to check some address(es) for membership in 172.16.0.0/16 but no addresses to check are given. I've filed `address` option should be mandatory · Issue #7 · logstash-plugins/logstash-filter-cidr · GitHub to make the address option mandatory.

If network =~ 192.168.0.([0-9]{0,3}) {

Almost:

if [network] =~ /^192\.168\.0\.([0-9]{0,3})/ {
4

Thanks for your help guys.
If I understand, I don't need to use any plugin to check the source network ? I don't find any documentation about that, this is great ! ( Even if it's with regex, which is absolutly not my best skill :slight_smile: )

I have try what you said, and my configuration file looks like to that :

filter {

	if [network] =~ /^172\.16\.([0-9]{0,3}).\([0-9]{0,3})/ {
		mutate {	
			add_tag => ["lan1"] 
		}
	}
	
	else if [network] =~ /^192\.168\.0\.([0-9]{0,3})/ {
		mutate {
			add_tag => ["lan2"] 
		}
	}
}

Logstash successfully started, but it will close up with this error code as soon as the first logs arrive :

RegexpError: unmatched close parenthesis: /^172.16.([0-9]{0,3}).([0-9]{0,3})/>

I don't see any unlosed parenthesis ? what's the matter ?

Thanks

5

If I understand, I don't need to use any plugin to check the source network ?

No, you don't have to but I'd expect regexp to be slower, less flexible, and less readable.

I don't find any documentation about that, this is great !

I don't see any unlosed parenthesis ?

Me neither. Reduce the expressions to narrow things down.

6

No, you don't have to but I'd expect regexp to be slower, less flexible, and less readable.

For sure !

Me neither. Reduce the expressions to narrow things down.

I've try with the /^172.16.0.([0-9]{0,3})/ network, and I still have the same error.
Anyway, If I reduce the expression, how can I manage any network with a non conventionnal netmask ? In my tests, the 172.16.0.0 network has a /16 netmask, and in my final environment I got all kind of netmask.

About what you said about the address option, I have too many host to check them one by one, that's why I try to manage them by they're network.

finaly, thanks for the documentation. I've already seen it but there is not much explaination about the regex

7

correction : /^172\.16\.0\.([0-9]{0,3})/ is better, but still same error.

8

About what you said about the address option, I have too many host to check them one by one, that's why I try to manage them by they're network.

But surely your events have a field containing the IP address you want to check against the CIDR pattern? In what you posted above the field appears to be named network.

9

That's why I wanted to use the cidr plugin. As I understand, this plugin check the host field and compare it with the network I indicate in the option.
By using regex, I thinks this is the same process but I can't be sure for the moment.

10

As I understand, this plugin check the host field

No, it checks the strings you tell it to check. If the IP address you want to check is in the host field you should use the filter like this:

cidr {
  address => ["%{host}"]
  network => ["172.16.0.0/16"]
  add_tag => ["lan1"]
}

See the examples in the documentation.

11

thanks for the example.
But you're right, the host field return the hostname and not the ip address. So Logstash can't find the source of the logs because I don't have any DNS server for my tries.
I will setting up a DNS server in my test network and I tell you back.

12

Well, I just noticed I just need to the hostname and the IPaddress of each hosts of my lan in /etc/hosts to resolving they're name.

[[main]>worker0] WARN logstash.filters.cidr - Invalid IP address, skipping {:address=>"%{host}", :event=>2017-01-27T14:00:40.894Z rsyslog Server listening on 0.0.0.0 port

If I understand, it try to interprat "%{host}" as a hostname ?

13

Here is a part of my config:

  if [host] =~ /10\.1\.([0-9]{1,3})/ {
    mutate { add_field => [ "city", "####" ] }
  }
  else if [host] =~ /10\.2\.([0-9]{1,3})/ {
    mutate { add_field => [ "city", "####" ] }
  }
  else if [host] =~ /10\.3\.([0-9]{1,3})/ {
    mutate { add_field => [ "city", "####" ] }
  }

I do this portion of my config, AFTER the primary parsing of the logs are done.

The CIDR plugin is doing the same thing.

Performing reverse DNS lookups is going to be even slower.

14

Thanks Kopacko for your example.
However, How can you treat VLSM networks ?

Thanks.

15

You can handle them just fine. My code is only looking at the octets.

If you wanted to get specific on say a /29:

if [host] =~ /10\.1\.([0-9]{1,3}).(0|1|2|3|4|5|6|7)/ {
  mutate { add_field => [ "city", "####" ] }
}

That code would cover 10.1.x.y/29. So any third octet and only the first /29 of the 4th octet.

16

If I understand, it try to interprat "%{host}" as a hostname ?

It tries to expand %{host} into the contents of the host field, but the event in question apparently doesn't have such a field so it leaves the string alone.

17

Ok, thanks for those explainations.
But as we discusting with magnusbaeck, Host fields are seems to be filled with hostnames and no with IP address.
This problem can be resolve with a reverse DNS but in any case, logstash have to resolve somethinks.
This is the containt of one of log which arrive :

 "severity" => "info",
     "@timestamp" => 2017-01-27T14:46:36.000Z,
       "@version" => "1",
           "host" => "client-log1",
    "programname" => "sshd",
         "procid" => "22984",
        "message" => " Server listening on :: port 22.",
           "type" => "rsyslog",
       "facility" => "authpriv"

Apparently, it can't resolve this name directly. It's seems to exist such plugin that do this : https://www.elastic.co/guide/en/logstash/current/plugins-filters-dns.html I will looking for that.
I have everythinks to resolv my problem now.
Thanks a lot for your help guys.

18

Ok.

This log message, is it syslog? Or what?

19

this log message come from a Rsyslog server which is send logs generated by CentOS

20

Gotcha.

I don't know much about Rsyslog, but what I do this:

Device sends it's log messages to a syslog-ng listener, who then forwards its up to a port that Logstash is listening on. I have syslog-ng NOT parse the message, which then captures the entire log message and appends the host IP to the header.

Here is an example of a log that I get:

<13>Jan 27 09:37:45.145416 x.x.x.x <3>Jan 27 09:37:45 HOSTNAME kernel: [71957.045063] iSCSI Login timeout on Network Portal x.x.x.x:3260