CIDR, location lookup in the most efficient way

[geertn444]

I want to map IPs in a certain range to a site. I don't want to map the IP ranges inside logstash config files, but in external files. CIDR has this possibility using network_path

So now i have a logic like this:

if [flow][dst_addr] {
cidr {
address => [ "%{[flow][dst_addr]}" ]
network_path => "/local/etc/subnets/a.txt"
add_field => { "[flow][dst_site]" => "A" }
}

cidr {
address => [ "%{[flow][dst_addr]}" ]
network_path => "/local/etc/subnets/B.txt"
add_field => { "[flow][dst_site]" => "B" }
}
cidr {
address => [ "%{[flow][dst_addr]}" ]
network_path => "/local/etc/subnets/C.txt"
add_field => { "[flow][dst_site]" => "C" }
}
cidr {
address => [ "%{[flow][dst_addr]}" ]
network_path => "/local/etc/subnets/D.txt"
add_field => { "[flow][dst_site]" => "D" }
}

The first disadvantage of this is that logstash will loop through ALL cidr matches, even when a match is found in the first CIDR filter. We should create a field or variable, called "found" which is false in the beginning, but once a match is found, it is set to true and all subsequent cidr matches are skipped. Is this possible with tags ?

The second disadvantage of this is that the smallest match isn't necesarrily returned.
For example, consider following example:

A.txt = 10.1.0.0/16
B.txt = 10.1.1.0/24

An ip 10.1.1.1 will match A and not do B anymore, while B is a more accurate match.
This is caused by the fact that our search is split across multiple files.
The best would be to create and look into a single file for all subnets.
Subnet searches could then be optimised from small to large for example.

10.1.1.0/24 : site B
10.1.0.0/16 : site A

the lookup can be ordered or optimised in this case by ordering subnets internally from /32 match (small) to largest blocks (/19, etc).

We could read multiple files at once, with something like:

cidr {
address => [ "%{[flow][dst_addr]}" ]
network_path => "/local/etc/subnets/A.txt"
network_path => "/local/etc/subnets/B.txt"
network_path => "/local/etc/subnets/C.txt"
#ordered from small to large in memory to prevent above problem
add_field => { "[flow][dst_site]" => "{%filename}" }
}

regards,
Geert

[magnusbaeck (Magnus Bäck)]

The first disadvantage of this is that logstash will loop through ALL cidr matches, even when a match is found in the first CIDR filter. We should create a field or variable, called "found" which is false in the beginning, but once a match is found, it is set to true and all subsequent cidr matches are skipped. Is this possible with tags ?

Yes. Add add_tag => "found" to you cidr filters and wrap each filter in if "found" not in [tags] { ... }.

We could read multiple files at once, with something like:

Why not just generate a single file that you feed a single cidr filter?

[geertn444]

Why not just generate a single file that you feed a single cidr filter?

Because i want to set a field differently based on which subnet is matched.
ie match on subnet A, B or C -> set field X
match on subnet D -> set field Y

or as in my example:
add_field => { "[flow][dst_site]" => "{%filename}" } <-- filename determines string that is set