Hi Guys,
I've spent more than a day trying to figure this out but just couldn't get it working. I'm using Filebeat to collect IIS logs, push them through Logstash and then into ES. Filebeat has the IIS module enabled and everything seems to be working until I attempt to enrich the entries with Geoip information. Lots of our requests are coming from the internal network. Geoip fails on every private IP address (obviously), but I need to map out internal requests as well, so I have to make the geoip plugin params conditional. Instead of forging the geoip info field-by-field, I've decided to simply change the source address to our external IP, that geoip can resolve. This worked like a charm but, unexpectedly, I ran into a concrete wall with the condition itself.
logstash.yml
xpack.monitoring.enabled: true
xpack.monitoring.elasticsearch.hosts: "http://SOMESERVER:9200"pipeline.id: beats
path.config: "../config/beats.conf"
pipeline.workers: 4
pipeline.batch.size: 500config.reload.automatic: true
queue.type: persisted
queue.max_bytes: 8gb
beats.conf
input {
beats {
port => 5044
}
}filter {
if "iis" in [tags] {
if [c-ip] =~ /^10./ or [c-ip] =~ /^192./ {
mutate { add_field => { "external-ip" => "xxx.xxx.xxx.xxx" } }
geoip {
source => "external-ip"
target => "geoip"
add_tag => [ "internal_request" ]
}
mutate { remove_field => [ "external-ip" ] }
}
else {
geoip {
source => "c-ip"
target => "geoip"
add_tag => [ "external_request" ]
}
}
}
}output {
if "iis" in [tags] {
elasticsearch {
hosts => ["http://SOMESERVER:9200"]
index => "iis-%{+YYYY.MM.dd}"
pipeline => "%{[@metadata][pipeline]}"
}
}
else if "event_log" in [tags] {
elasticsearch {
hosts => ["http://SOMESERVER:9200"]
index => "event_log-%{+YYYY.MM.dd}"
}
}
else if "elasticsearch_log" in [tags] {
elasticsearch {
hosts => ["http://SOMESERVER:9200"]
index => "elasticsearch_log-%{+YYYY.MM.dd}"
}
}
else if [@metadata][pipeline] {
elasticsearch {
hosts => ["http://SOMESERVER:9200"]
index => "unidentified-%{+YYYY.MM.dd}"
}
}
else {
elasticsearch {
hosts => ["http://SOMESERVER:9200"]
index => "other-%{+YYYY.MM.dd}"
}
}
}
The problem is that the IF statement always falls into the ELSE path, not matter what I do.
I've tried simple regex:
if [c-ip] =~ /^10./ or [c-ip] =~ /^192./ {
...
I've tried complex regex:
if [c-ip] =~ /^((127.)|(10.)|(172.1[6-9].)|(172.2[0-9].)|(172.3[0-1].)|(192.168.))/ {
...
...these all land in the ELSE path.
I've tried using CIDR as someone recommended here on the forum:
cidr { address => [ "%{c-ip}" ] network => [ "10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16", "127.0.0.0/8" ] add_field => { "source.network" => "internal" } } ...
...or:
cidr {
address => [ "[c-ip]" ]
network => [ "10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16", "127.0.0.0/8" ]
add_field => { "source.network" => "internal" }
}
...
...or:
cidr {
address => [ "c-ip" ]
network => [ "10.0.0.0/8", "172.16.0.0/12", "192.168.0.0/16", "127.0.0.0/8" ]
add_field => { "source.network" => "internal" }
}
...
They all failed with:
[WARN ][logstash.filters.cidr ][beats] Invalid IP address, skipping...
This was the point where started thinking that maybe it's not my regex or the CIDR call that is faulty, but "c-ip" simply isn't available as a field so I'm comparing NULL every time.
However, this works just fine (although it's not conditional as I'd like):
...
filter {
geoip {
source => "c-ip"
target => "geoip"
add_tag => [ "geoip" ]
}
}
...
...so "c-ip" is in fact available in some form or shape, just not directly from the filter plugin.
I've also tried [source][address] and [source][ip] none of them seemed to have worked. Without these, I can't filter in my conditional and can't utilize geoip properly.
So my question is, how can I access the source IP of an IIS request from within the filter section of a config file. If someone could answer this, it would really make my New Year 
Cheers,
Laz