Mapping ip address to specific network

Elastic Stack Logstash
1

Hello,

I have a problem preparing logstash script.
I'm parsing a syslog, then I get a source IPv4 address. I would like to match this addres to a networks defined in file and add a field with network name.

example of network definition:

193.254.136.0/26; Network 1
212.23.99.0/24; Network 2
160.218.40.128/25; Network 3

for example when I get address 193.254.136.6 I would like to add field containing "Network 1"
For address i.e. 212.23.99.14 I would like to have this field with "Network 2" value.

I've checked possibility of defining network as regular expression however the problem is number of entries. I have like 3k+ networks with different masks and preparing RE for each seems impossible.

Second option (also abandoned) was to expand the networks to all hosts which they contains.
Then use translate filter plugin with dictionary_path to file containg IPs and net name but it result in 800k+ lines dictionary :slight_smile:
I'm affraid of performance issues

Gone through geoip and cidr filters with no concept found (cidr seemed promising but it only gives info if ip is in range of the listed networks, without info in which one)

Any ideas?
Thanks in advance

Michal

2

logstash filter add_field

3

Thanks for reply
add_field yes but how to differentiate values depending on Ip address using this filter?

4

change the 193.xx.xx/26 to two value, one is ip one is netmask if value2 = 26 add network1 filed.

5

i don't remeber if is have include .
if [ "value"] =~ [" 193.254"] i don't know it ok or not ,include
@mmalewicz

6

It seems like the cidr filter might be the way to go as it allows you to specify and match ip range. It will however require one filter per network, and as you have a large number of them it could get slow. You may however be able divide your processing into group so that you do not have to process all combinations for every entry. It will still result in a large config, but something along these lines might work:

input {
  generator {
    lines => ['193.254.136.0','212.23.99.0','160.218.40.128']
    count => 1
  } 
} 

filter {
    dissect {
        mapping => {
            "message" => "%{ip}.%{+ip}.%{}"
      }
    }

    if [ip] == "193.254" {
        cidr {
            add_field => { "network" => "Network 1" }
            address => [ "%{message}" ]
            network => [ "193.254.136.0/26" ]
        }
        cidr {
            add_field => { "network" => "Network X" }
            address => [ "%{message}" ]
            network => [ "193.254.1.0/26" ]
        }
    } else if [ip] == "212.23" {
        cidr {
            add_field => { "network" => "Network 2" }
            address => [ "%{message}" ]
            network => [ "212.23.99.0/24" ]
        }
    } else if [ip] == "160.218" {
        cidr {
            add_field => { "network" => "Network 3" }
            address => [ "%{message}" ]
            network => [ "160.218.40.128/25" ]
        }
    }
}

output {
  stdout { codec => rubydebug }
}

I would also test the large expanded dictionary file for the translate plugin to see what level of performance you get, as it would result in a much more compact configuration.

7

Thanks, I will test large dictionary perfomance as network definition may change over the time and I will just change the dictionary, not the code

8

One other candidate could also be the jdbc static filter, based on a table with expanded IP-network mappings.

9

Expanded network 800k+ dictionary + translate filter plugin did the job. Thanks for all hints :slight_smile:

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.