# Logstash 2.3 - GeoIP problem

**URL:** <https://discuss.elastic.co/t/logstash-2-3-geoip-problem/104974>\
**Category:** Logstash\
**Created:** [October 23, 2017, 9:28pm UTC](https://discuss.elastic.co/t/logstash-2-3-geoip-problem/104974 "2017-10-23T21:28:29Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![gh0stid](https://avatars.discourse-cdn.com/v4/letter/g/f9ae1b/32.png) [@gh0stid](https://discuss.elastic.co/u/gh0stid)\
**Post date:** [October 23, 2017, 9:28pm UTC](https://discuss.elastic.co/t/logstash-2-3-geoip-problem/104974/1 "2017-10-23T21:28:29Z")

</div>

Hello people.. sorry to bother , once again

ALL of my config is working and I mean it..  
absolutely all of it. mostely ripped code from siemonster and other places, but all together " work "

EXCEPT GeoIP

Please gimme a hand, heres the config :

```
filter {
  if [type] == "syslog" {
      if "devname" in [message] {
      mutate {
      add_tag => ["COUPEFEU", "FORTIGATE"]
      }
        }

      if "%ASA-" in [message] {
      mutate {
      add_tag => ["Firewall", "ASA"]
      }
	  }
      if "VPN" in [message] {
      mutate {
      add_tag => ["VPN"]
      }
	}
	  if "SOC" in [message] {
      mutate {
      add_tag => ["SOC"]
      }
	} 
      if "IPS" in [message] {
      mutate {
      add_tag => ["IPS"]
      }
           }
      if "printer" in [message] {
      mutate {
      add_tag => ["hp-printers"]
           }
      }

#
#
#
# Parse Fortigate
if "FORTIGATE" in [tags] {
grok {
  match => ["message", "%{SYSLOG5424PRI}%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_host} %{GREEDYDATA:kv}"]
  remove_field => ["message"]
  remove_field => ["syslog_timestamp"]
# remove_field => ["type"]
}
syslog_pri { }

kv {
      source => "kv"
      exclude_keys => ["type", "subtype"]
      field_split => " "
      value_split => "="
}

date {
  match => ["logtimestamp", "ISO8601"]
  locale => "en"
  timezone =>"America/Montreal"
  remove_field => ["logtimestamp"]
}

mutate {
      convert => ["rcvdbyte", "integer"]
      convert => ["countdlp", "integer"]
      convert => ["countweb", "integer"]
      convert => ["countav", "integer"]
      convert => ["countemail", "integer"]
      convert => ["countips", "integer"]
      convert => ["duration", "integer"]
      convert => ["sentpkt", "integer"]
      convert => ["rcvdpkt", "integer"]
      convert => ["sentbyte", "integer"]
      convert => ["shaperdroprcvdbyte", "integer"]
      convert => ["shaperdropsentbyte", "integer"]
      convert => ["filesize", "integer"]
      convert => ["count", "integer"]
      convert => ["total", "integer"]
      convert => ["totalsession", "integer"]
      convert => ["bandwidth", "integer"]
      #rename => { "type" => "ftg-type" }
    }

#Geolocate logs that have SourceAddress and if that SourceAddress is a non-RFC1918 address or APIPA address
if [srcip] and [srcip] !~ "(^127\.0\.0\.1)|(^10\.)|(^172\.1[6-9]\.)|(^172\.2[0-9]\.)|(^172\.3[0-1]\.)|(^192\.168\.)|(^169\.254\.)" {
    geoip {
         database => "/etc/logstash/GeoLiteCity.dat"
         source => "srcip"
         target => "SourceGeo"
         add_tag => ["traffic-wan"]
    }
}

#filtrer le traffic RITM du traffic internet pour input dans un second index
if [srcip] and [srcip] =~ "(^127\.0\.0\.1)|(^10\.)|(^172\.1[6-9]\.)|(^172\.2[0-9]\.)|(^172\.3[0-1]\.)|(^192\.168\.)|(^169\.254\.)" {
    mutate {
         add_tag => ["src-traffic-ritm"]
         ["SourceGeo.location"] => "geo_point"
    }

    #Delete 0,0 in SourceGeo.location if equal to 0,0
    #if ([srcip.location] and [srcip.location] =~ "0,0") {
      #mutate {
       # ["SourceGeo.location"] => "geo_point"
      #}
    #}
  #}

#Geolocate logs that have DestinationAddress and if that DestinationAddress is a non-RFC1918 address or APIPA address
if [dstip] and [dstip] !~ "(^127\.0\.0\.1)|(^10\.)|(^172\.1[6-9]\.)|(^172\.2[0-9]\.)|(^172\.3[0-1]\.)|(^192\.168\.)|(^169\.254\.)" {
    geoip {
         database => "/etc/logstash/GeoLiteCity.dat"
         source => "dstip"
         target => "DestinationGeo"
         add_tag => ["traffic-wan"]
    }
}

#filtrer le traffic RITM du traffic internet pour input dans un second index
if [dstip] and [dstip] =~ "(^127\.0\.0\.1)|(^10\.)|(^172\.1[6-9]\.)|(^172\.2[0-9]\.)|(^172\.3[0-1]\.)|(^192\.168\.)|(^169\.254\.)" {
    mutate {
         add_tag => ["dst-traffic-ritm"]
         ["DestinationGeo.location"] => "geo_point"
    }

    #Delete 0,0 in DestinationGeo.location if equal to 0,0
    #if ([dstip.location] and [dstip.location] =~ "0,0") {
      #mutate {
       # ["DestinationGeo.location"] => "geo_point"
        # }
       #}
     }
  }
#
#
#
#

```

STRIPPED LACK OF SPACE - pastebin here : [https://pastebin.com/4DnmxDa1](https://pastebin.com/4DnmxDa1)

somehow, it just.. doesnt populate the geoip.. dont know why nor where to look for an answer...  
Thank you .. and sorry to bother :\

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [October 24, 2017, 5:19am UTC](https://discuss.elastic.co/t/logstash-2-3-geoip-problem/104974/2 "2017-10-24T05:19:07Z")

</div>

Please show an example event, preferably using a `stdout { codec => rubydebug }` output.

---

<div class="post-metadata">

**Author:** ![gh0stid](https://avatars.discourse-cdn.com/v4/letter/g/f9ae1b/32.png) [@gh0stid](https://discuss.elastic.co/u/gh0stid)\
**Post date:** [October 24, 2017, 7:37pm UTC](https://discuss.elastic.co/t/logstash-2-3-geoip-problem/104974/3 "2017-10-24T19:37:56Z")

</div>

Hello

heres an exemple. please see pastebin :

[https://pastebin.com/1rcjH2jS](https://pastebin.com/1rcjH2jS)

Thank you.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [October 25, 2017, 5:20am UTC](https://discuss.elastic.co/t/logstash-2-3-geoip-problem/104974/4 "2017-10-25T05:20:32Z")

</div>

And what isn't working? The `DestinationGeo` field appears to be populated just fine. The `SourceGeo` isn't, but that's expected with the 10.0.0.0/8 address in `srcip`.

---

<div class="post-metadata">

**Author:** ![gh0stid](https://avatars.discourse-cdn.com/v4/letter/g/f9ae1b/32.png) [@gh0stid](https://discuss.elastic.co/u/gh0stid)\
**Post date:** [October 25, 2017, 5:33am UTC](https://discuss.elastic.co/t/logstash-2-3-geoip-problem/104974/5 "2017-10-25T05:33:22Z")

</div>

Im not at the office right now and tomorrow but I just cant get a tile map to work.

Theres no "field" geo

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [October 25, 2017, 5:43am UTC](https://discuss.elastic.co/t/logstash-2-3-geoip-problem/104974/6 "2017-10-25T05:43:01Z")

</div>

You mean Kibana doesn't list any geo\_point fields to build your tile map from? If so you'll have to adjust your index template so that additional fields have the geo\_point. Read up on mappings and index templates in the ES documentation.

---

<div class="post-metadata">

**Author:** ![gh0stid](https://avatars.discourse-cdn.com/v4/letter/g/f9ae1b/32.png) [@gh0stid](https://discuss.elastic.co/u/gh0stid)\
**Post date:** [October 25, 2017, 5:55am UTC](https://discuss.elastic.co/t/logstash-2-3-geoip-problem/104974/7 "2017-10-25T05:55:01Z")

</div>

Geo.point is present on the kibana visualisation  
Yet it is empty

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [October 25, 2017, 6:07am UTC](https://discuss.elastic.co/t/logstash-2-3-geoip-problem/104974/8 "2017-10-25T06:07:42Z")

</div>

I don't know what you mean. Show a screenshot and what the index's mappings look like (use ES's get mapping API).

---

<div class="post-metadata">

**Author:** ![gh0stid](https://avatars.discourse-cdn.com/v4/letter/g/f9ae1b/32.png) [@gh0stid](https://discuss.elastic.co/u/gh0stid)\
**Post date:** [October 26, 2017, 4:56pm UTC](https://discuss.elastic.co/t/logstash-2-3-geoip-problem/104974/9 "2017-10-26T16:56:50Z")

</div>

![image](https://us1.discourse-cdn.com/elastic/original/3X/8/1/81430fa1c464ca5f1edfe70aac1dcde9b3c5d319.png)

Heres the screenshot as requested.

and the mapping :

> {  
> "order": 0,  
> "template": "logstash-_",  
> "settings": {  
> "index": {  
> "refresh\_interval": "5s"  
> }  
> },  
> "mappings": {  
> "default": {  
> "dynamic\_templates": [  
> {  
> "message\_field": {  
> "mapping": {  
> "fielddata": {  
> "format": "disabled"  
> },  
> "index": "analyzed",  
> "omit\_norms": true,  
> "type": "string"  
> },  
> "match\_mapping\_type": "string",  
> "match": "message"  
> }  
> },  
> {  
> "string\_fields": {  
> "mapping": {  
> "fielddata": {  
> "format": "disabled"  
> },  
> "index": "analyzed",  
> "omit\_norms": true,  
> "type": "string",  
> "fields": {  
> "raw": {  
> "ignore\_above": 256,  
> "index": "not\_analyzed",  
> "type": "string"  
> }  
> }  
> },  
> "match\_mapping\_type": "string",  
> "match": "_"  
> }  
> }  
> ],  
> "\_all": {  
> "omit\_norms": true,  
> "enabled": true  
> },  
> "properties": {  
> "@timestamp": {  
> "type": "date"  
> },  
> "geoip": {  
> "dynamic": true,  
> "properties": {  
> "ip": {  
> "type": "ip"  
> },  
> "latitude": {  
> "type": "float"  
> },  
> "location": {  
> "type": "geo\_point"  
> },  
> "longitude": {  
> "type": "float"  
> }  
> }  
> },  
> "@version": {  
> "index": "not\_analyzed",  
> "type": "string"  
> }  
> }  
> }  
> },  
> "aliases": {}  
> }

---

<div class="post-metadata">

**Author:** ![gh0stid](https://avatars.discourse-cdn.com/v4/letter/g/f9ae1b/32.png) [@gh0stid](https://discuss.elastic.co/u/gh0stid)\
**Post date:** [October 26, 2017, 5:10pm UTC](https://discuss.elastic.co/t/logstash-2-3-geoip-problem/104974/10 "2017-10-26T17:10:45Z")

</div>

Ive just modified it to match ... ( at least I think ) we will see tomorrow on index-creation

```
{
  "order": 0,
  "template": "logstash-*",
  "settings": {
    "index": {
      "refresh_interval": "5s"
    }
  },
  "mappings": {
    "_default_": {
      "dynamic_templates": [
        {
          "message_field": {
            "mapping": {
              "fielddata": {
                "format": "disabled"
              },
              "index": "analyzed",
              "omit_norms": true,
              "type": "string"
            },
            "match_mapping_type": "string",
            "match": "message"
          }
        },
        {
          "string_fields": {
            "mapping": {
              "fielddata": {
                "format": "disabled"
              },
              "index": "analyzed",
              "omit_norms": true,
              "type": "string",
              "fields": {
                "raw": {
                  "ignore_above": 256,
                  "index": "not_analyzed",
                  "type": "string"
                }
              }
            },
            "match_mapping_type": "string",
            "match": "*"
          }
        }
      ],
      "_all": {
        "omit_norms": true,
        "enabled": true
      },
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "geoip": {
          "dynamic": true,
          "properties": {
            "ip": {
              "type": "ip"
            },
            "latitude": {
              "type": "float"
            },
            "location": {
              "type": "geo_point"
            },
            "longitude": {
              "type": "float"
            }
          }
        },
        "DestinationGeo": {
          "dynamic": true,
          "properties": {
            "ip": {
              "type": "ip"
            },
            "latitude": {
              "type": "float"
            },
            "location": {
              "type": "geo_point"
            },
            "longitude": {
              "type": "float"
            }
          }
        },
        "SourceGeo": {
          "dynamic": true,
          "properties": {
            "ip": {
              "type": "ip"
            },
            "latitude": {
              "type": "float"
            },
            "location": {
              "type": "geo_point"
            },
            "longitude": {
              "type": "float"
            }
          }
        },
        "@version": {
          "index": "not_analyzed",
          "type": "string"
        }
      }
    }
  },
  "aliases": {}
}
```

---

<div class="post-metadata">

**Author:** ![gh0stid](https://avatars.discourse-cdn.com/v4/letter/g/f9ae1b/32.png) [@gh0stid](https://discuss.elastic.co/u/gh0stid)\
**Post date:** [October 27, 2017, 5:27pm UTC](https://discuss.elastic.co/t/logstash-2-3-geoip-problem/104974/11 "2017-10-27T17:27:08Z")

</div>

unfortunately ..

still not working :\

dang

Any help please ? im sure its something really simple... probably with my mapping..

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [October 27, 2017, 5:28pm UTC](https://discuss.elastic.co/t/logstash-2-3-geoip-problem/104974/12 "2017-10-27T17:28:27Z")

</div>

Please show a) an example document and b) the index mappings ( **not the index template** ).

---

<div class="post-metadata">

**Author:** ![gh0stid](https://avatars.discourse-cdn.com/v4/letter/g/f9ae1b/32.png) [@gh0stid](https://discuss.elastic.co/u/gh0stid)\
**Post date:** [October 27, 2017, 5:31pm UTC](https://discuss.elastic.co/t/logstash-2-3-geoip-problem/104974/13 "2017-10-27T17:31:27Z")

</div>

Elastic HQ :

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/c/b/cb42a8a4936aaf91ca8c2575c84bf0e34eae069d.png)

 ![image](https://us1.discourse-cdn.com/elastic/original/3X/e/3/e30ac8dbc0747695c1ffed67dfe1cf0322ecc8e8.png)

mapping : [https://pastebin.com/2v87MK7H](https://pastebin.com/2v87MK7H)

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [October 30, 2017, 6:35am UTC](https://discuss.elastic.co/t/logstash-2-3-geoip-problem/104974/14 "2017-10-30T06:35:49Z")

</div>

Okay, so both `[SourceGeo][location]` and `[DestinationGeo][location]` have been mapped as geo\_point. Are those field still not available in Kibana?

---

<div class="post-metadata">

**Author:** ![gh0stid](https://avatars.discourse-cdn.com/v4/letter/g/f9ae1b/32.png) [@gh0stid](https://discuss.elastic.co/u/gh0stid)\
**Post date:** [October 30, 2017, 6:50am UTC](https://discuss.elastic.co/t/logstash-2-3-geoip-problem/104974/15 "2017-10-30T06:50:42Z")

</div>

Negative.

They are not in kabana.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [October 30, 2017, 6:55am UTC](https://discuss.elastic.co/t/logstash-2-3-geoip-problem/104974/16 "2017-10-30T06:55:12Z")

</div>

Did you refresh the field list in Kibana?

---

<div class="post-metadata">

**Author:** ![gh0stid](https://avatars.discourse-cdn.com/v4/letter/g/f9ae1b/32.png) [@gh0stid](https://discuss.elastic.co/u/gh0stid)\
**Post date:** [October 30, 2017, 7:07am UTC](https://discuss.elastic.co/t/logstash-2-3-geoip-problem/104974/17 "2017-10-30T07:07:16Z")

</div>

Yes I did.  
Right now I am not at the office. But tomorrow in about 6-7h ill be

Ill be able to post another screenshot of the issue.

---

<div class="post-metadata">

**Author:** ![gh0stid](https://avatars.discourse-cdn.com/v4/letter/g/f9ae1b/32.png) [@gh0stid](https://discuss.elastic.co/u/gh0stid)\
**Post date:** [October 31, 2017, 5:20pm UTC](https://discuss.elastic.co/t/logstash-2-3-geoip-problem/104974/18 "2017-10-31T17:20:45Z")

</div>

![image](https://us1.discourse-cdn.com/elastic/original/3X/9/d/9dc05b928e4d0705ba131d2e35ff0c5f117be639.png)

heres the "field" in kibana when trying to visualize

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [October 31, 2017, 7:59pm UTC](https://discuss.elastic.co/t/logstash-2-3-geoip-problem/104974/19 "2017-10-31T19:59:47Z")

</div>

Please show:

- An example document. Copy/paste from Kibana's JSON tab.
- The index mappings. Use the get mapping API. No screenshot.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 28, 2017, 7:59pm UTC](https://discuss.elastic.co/t/logstash-2-3-geoip-problem/104974/20 "2017-11-28T19:59:47Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.