# How to filter out internal IPs and localhost.localdomain for geoip?

**URL:** <https://discuss.elastic.co/t/how-to-filter-out-internal-ips-and-localhost-localdomain-for-geoip/84184>\
**Category:** Logstash\
**Created:** [May 1, 2017, 9:35pm UTC](https://discuss.elastic.co/t/how-to-filter-out-internal-ips-and-localhost-localdomain-for-geoip/84184 "2017-05-01T21:35:13Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![dfinn](https://avatars.discourse-cdn.com/v4/letter/d/8c91f0/32.png) [@dfinn](https://discuss.elastic.co/u/dfinn)\
**Post date:** [May 1, 2017, 9:35pm UTC](https://discuss.elastic.co/t/how-to-filter-out-internal-ips-and-localhost-localdomain-for-geoip/84184/1 "2017-05-01T21:35:13Z")

</div>

My logs are full of errors like so:

[2017-05-01T15:11:18,901][ERROR][logstash.filters.geoip] IP Field contained invalid IP address or hostname {:exception=\>java.net.UnknownHostException: localhost.localdomain: Name or service not known, :field=\>"clientip", :event=\>2017-05-01T21:11:13.000Z [ps-prod-app07.domain.com](http://ps-prod-app07.domain.com) localhost.localdomain - - [01/May/2017:21:11:13 +0000] "GET /server-status HTTP/1.0" 200 28786 "-" "Lynx/2.8.6rel.5 libwww-FM/2.14 SSL-MM/1.4.1 OpenSSL/1.0.0-fips"}

As best as I can tell, we are also getting geoip issues on all our logs which have internal 10.x.x.x IPs in clientip. What's the best way to filter these out. Here's our current config for the geoip stuff:

```
filter {
  if [type] == "apache" {
grok {
  match => ["message", "%{COMBINEDAPACHELOG}"]
}
date {
  match => ["timestamp" , "dd/MMM/yyyy:HH:mm:ss Z"]
}
if [agent] != "-" {
  useragent {
    source => "agent"
  }
}
geoip {
  source => "clientip"
  target => "geoip"
  database => "/etc/logstash/GeoLiteCity.dat"
  add_field => ["[geoip][coordinates]", "%{[geoip][longitude]}" ]
  add_field => ["[geoip][coordinates]", "%{[geoip][latitude]}" ]
}
mutate {
  convert => ["[geoip][coordinates]", "float"]
}
}
}

```

I tried wrapping a simple if statement that said "if != localhost.localdomain" around the geoip stanza but that didn't seem to work.

What I'd like to do is have geoip not run if it's either localhost.localdomain or an 10.x.x.x IP but I'm not sure the best/cleanest way to accomplish that.

Any help would be much appreciated.

---

<div class="post-metadata">

**Author:** ![jkuang](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jkuang/32/72637_2.png) [@jkuang](https://discuss.elastic.co/u/jkuang)\
**Post date:** [May 1, 2017, 10:16pm UTC](https://discuss.elastic.co/t/how-to-filter-out-internal-ips-and-localhost-localdomain-for-geoip/84184/2 "2017-05-01T22:16:42Z")

</div>

You can use drop{} .

[https://www.elastic.co/guide/en/logstash/current/plugins-filters-drop.html](https://www.elastic.co/guide/en/logstash/current/plugins-filters-drop.html)

---

<div class="post-metadata">

**Author:** ![dfinn](https://avatars.discourse-cdn.com/v4/letter/d/8c91f0/32.png) [@dfinn](https://discuss.elastic.co/u/dfinn)\
**Post date:** [May 1, 2017, 10:18pm UTC](https://discuss.elastic.co/t/how-to-filter-out-internal-ips-and-localhost-localdomain-for-geoip/84184/3 "2017-05-01T22:18:35Z")

</div>

I'm not sure that's what I want. I don't want to lose these logs, they still have valuable information in them. I just don't want the geoip filter applied to them.

---

<div class="post-metadata">

**Author:** ![jkuang](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jkuang/32/72637_2.png) [@jkuang](https://discuss.elastic.co/u/jkuang)\
**Post date:** [May 1, 2017, 10:26pm UTC](https://discuss.elastic.co/t/how-to-filter-out-internal-ips-and-localhost-localdomain-for-geoip/84184/4 "2017-05-01T22:26:07Z")

</div>

Then just use an if statement.

Pseudo code:

Parse the "message" into smaller variables.

If IP !=10.x.x.x.x or HOST\_NAME !=localhost.localdomain  
geoip  
else  
{}  
.....

---

<div class="post-metadata">

**Author:** ![dfinn](https://avatars.discourse-cdn.com/v4/letter/d/8c91f0/32.png) [@dfinn](https://discuss.elastic.co/u/dfinn)\
**Post date:** [May 1, 2017, 10:27pm UTC](https://discuss.elastic.co/t/how-to-filter-out-internal-ips-and-localhost-localdomain-for-geoip/84184/5 "2017-05-01T22:27:40Z")

</div>

As I mentioned in my original post, I tried that and it didn't work so I was wondering if either I'm not doing something right or there is a better way of doing it.

---

<div class="post-metadata">

**Author:** ![jkuang](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jkuang/32/72637_2.png) [@jkuang](https://discuss.elastic.co/u/jkuang)\
**Post date:** [May 1, 2017, 10:56pm UTC](https://discuss.elastic.co/t/how-to-filter-out-internal-ips-and-localhost-localdomain-for-geoip/84184/6 "2017-05-01T22:56:53Z")

</div>

I couldn't tell that's what you did as your if statement post does not have a variable for the comparison.

"if != localhost.localdomain"

vs mine

If IP !=10.x.x.x.x or HOST\_NAME !=localhost.localdomain

---

<div class="post-metadata">

**Author:** ![dfinn](https://avatars.discourse-cdn.com/v4/letter/d/8c91f0/32.png) [@dfinn](https://discuss.elastic.co/u/dfinn)\
**Post date:** [May 1, 2017, 11:00pm UTC](https://discuss.elastic.co/t/how-to-filter-out-internal-ips-and-localhost-localdomain-for-geoip/84184/7 "2017-05-01T23:00:28Z")

</div>

Sorry, that was a typo.

It was:

```
if clientip != localhost.localdomain

```

I'll have to test again but I believe this caused a stop to all geoip filtering. Also, I'm not totally sure how to match 10.x.x.x internal IPs. Can I use regex in those if statements? Something like:

```
if clientip != (10\.[0-9]+\.[0-9]+\.[0-9]+|localhost\.localdomain)

```

would that work?

---

<div class="post-metadata">

**Author:** ![jkuang](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jkuang/32/72637_2.png) [@jkuang](https://discuss.elastic.co/u/jkuang)\
**Post date:** [May 1, 2017, 11:33pm UTC](https://discuss.elastic.co/t/how-to-filter-out-internal-ips-and-localhost-localdomain-for-geoip/84184/8 "2017-05-01T23:33:39Z")

</div>

I think you need [] , e.g [clientip] and [HOST\_NAME].

Here is my working example.

```auto
filter{
grok {
    match => ["message", "%{HOSTNAME:hostname}"]
  }

if [hostname]=="elastic.search.com" {
 mutate{ remove_field =>"@version"}

}

}

```

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [May 3, 2017, 7:45am UTC](https://discuss.elastic.co/t/how-to-filter-out-internal-ips-and-localhost-localdomain-for-geoip/84184/9 "2017-05-03T07:45:23Z")

</div>

> Can I use regex in those if statements?

Yes, if you use the `=~` operator. See examples in the documentation.

> **[Accessing event data and fields | Logstash Reference \[8.11\] | Elastic](https://www.elastic.co/guide/en/logstash/current/event-dependent-configuration.html)**

You can also use a cidr filter.

---

<div class="post-metadata">

**Author:** ![dfinn](https://avatars.discourse-cdn.com/v4/letter/d/8c91f0/32.png) [@dfinn](https://discuss.elastic.co/u/dfinn)\
**Post date:** [May 3, 2017, 5:29pm UTC](https://discuss.elastic.co/t/how-to-filter-out-internal-ips-and-localhost-localdomain-for-geoip/84184/10 "2017-05-03T17:29:35Z")

</div>

Thanks @magnusbaeck, I ended up using the cidr filter. My guess is that this could be cleaned up (so feel free to offer suggestions) but this seems to be working now to filter out any internal 10.x.x.x IPs or anytime that either localhost or localhost.localdomain ends up in clientip

```
filter {
  if [type] == "apache" {
grok {
  match => ["message", "%{COMBINEDAPACHELOG}"]
}
date {
  match => ["timestamp" , "dd/MMM/yyyy:HH:mm:ss Z"]
}
if [agent] != "-" {
  useragent {
    source => "agent"
  }
}
# first check if clientip field even exists
if [clientip] {
  # then check if it's in the 10.x.x.x space and if it is add the internalIP tag
  if [clientip] !~ /localhost|\-/ {
    cidr {
      add_tag => ["internalIP"]
      address => ["%{clientip}"]
      network => ["10.0.0.0/8", "127.0.0.1"]
    }
  }
  # don't run geoip if it's internalIP or localhost (aka only external IPs)
  if "internalIP" not in [tags] and [clientip] !~ /localhost|\-/ {
    geoip {
      source => "clientip"
      target => "geoip"
      database => "/etc/logstash/GeoLiteCity.dat"
      add_field => ["[geoip][coordinates]", "%{[geoip][longitude]}" ]
      add_field => ["[geoip][coordinates]", "%{[geoip][latitude]}" ]
    }
    mutate {
      convert => ["[geoip][coordinates]", "float"]
    }
  }
}
  }
}
```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [May 31, 2017, 5:30pm UTC](https://discuss.elastic.co/t/how-to-filter-out-internal-ips-and-localhost-localdomain-for-geoip/84184/11 "2017-05-31T17:30:12Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.