# GEOIP / Nginx logs, no drop down in map visual

**URL:** <https://discuss.elastic.co/t/geoip-nginx-logs-no-drop-down-in-map-visual/91106>\
**Category:** Kibana\
**Created:** [June 28, 2017, 1:02pm UTC](https://discuss.elastic.co/t/geoip-nginx-logs-no-drop-down-in-map-visual/91106 "2017-06-28T13:02:23Z")\
**Posts on this page:** 20\
**Page:** 1

<div class="post-metadata">

**Author:** ![runtman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/runtman/32/18136_2.png) [@runtman](https://discuss.elastic.co/u/runtman)\
**Post date:** [June 28, 2017, 1:02pm UTC](https://discuss.elastic.co/t/geoip-nginx-logs-no-drop-down-in-map-visual/91106/1 "2017-06-28T13:02:23Z")

</div>

Hello, so I want to visualise some of the Nginx logs we have with the geoip source.

So I have the following in Filebeat configurations to push the logs

```
- input_type: log
  paths:
    - /var/log/nginx/dev/access.log
  document_type: nginx_dev_access

```

Then in my nginx filter, I have the following GROK and filters:

```
filter {
        if [type] == "nginx_dev_access" {
                grok {
                        match => { "message" => "%{COMBINEDAPACHELOG}" }
                }
                geoip {
                        source => "clientip"
                        target => "geoip"
                        database => "/etc/logstash/GeoLiteCity.dat"
                        add_field => ["[geoip][coordinates]", "%{[geoip][longitude]}" ]
                        add_field => ["[geoip][coordinates]", "%{[geoip][latitude]}" ]
                }
                mutate {
                        convert => ["[geoip][coordinates]", "float"]
                }
        }
 }

```

If I go to the "Discover" home screen and do a search, I can see logs with these filters (image attached)

 ![](https://us1.discourse-cdn.com/elastic/original/3X/d/7/d7090f647a9b624c4c90e10be22f29a619588aaf.png)

I see the available fields, they appear to be mark as unknown (the small ?)

So if I save this search and go to plot a map in visulations, I only have these options.

 ![](https://us1.discourse-cdn.com/elastic/original/3X/a/9/a9b1979ac9cd0453bdd6fd241bad8abbee31fed2.png)

How do I get my custom filters to appear in that list?

It may be worth mentioning, when this ELK stack was built the default filebeats dashboards were adding thinking they would work, they do not as it goes through Logstash first.

---

<div class="post-metadata">

**Author:** ![runtman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/runtman/32/18136_2.png) [@runtman](https://discuss.elastic.co/u/runtman)\
**Post date:** [June 28, 2017, 1:32pm UTC](https://discuss.elastic.co/t/geoip-nginx-logs-no-drop-down-in-map-visual/91106/2 "2017-06-28T13:32:41Z")

</div>

Following other threads, I have reduced the filter to just this:

```
            geoip {
                    source => "clientip"
            }

```

I can again see data under geoip.ip but my drop down only has the 4 options, I'm assuming loading some template has meddled with that drop down box?

I have noticed the mapping is set to "keyword"

```
root@ip-172-31-25-192:/var/log# curl -u elastic:changeme http://172.31.18.116:9200/filebeat*/_mapping/nginx_dev_access/field/geoip.ip?pretty
{
  "filebeat-2017.06.28" : {
    "mappings" : {
      "nginx_dev_access" : {
        "geoip.ip" : {
          "full_name" : "geoip.ip",
          "mapping" : {
            "ip" : {
              "type" : "keyword",
              "ignore_above" : 1024
            }
          }
        }
      }
    }
  }
}

```

This needs to be geo\_point I am guessing? Even if I empty logstash and elastic search ,load the default filebeat template. This data comes back as "keyword" is there anyway I can inforce it always be geo\_point

---

<div class="post-metadata">

**Author:** ![runtman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/runtman/32/18136_2.png) [@runtman](https://discuss.elastic.co/u/runtman)\
**Post date:** [June 29, 2017, 9:05am UTC](https://discuss.elastic.co/t/geoip-nginx-logs-no-drop-down-in-map-visual/91106/3 "2017-06-29T09:05:35Z")

</div>

Bump

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [June 29, 2017, 10:15am UTC](https://discuss.elastic.co/t/geoip-nginx-logs-no-drop-down-in-map-visual/91106/4 "2017-06-29T10:15:34Z")

</div>

What's the mapping for the `geoip.location` field, not the `geoip.ip` field.

---

<div class="post-metadata">

**Author:** ![runtman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/runtman/32/18136_2.png) [@runtman](https://discuss.elastic.co/u/runtman)\
**Post date:** [June 30, 2017, 9:53am UTC](https://discuss.elastic.co/t/geoip-nginx-logs-no-drop-down-in-map-visual/91106/5 "2017-06-30T09:53:45Z")

</div>

> [@runtman](#):
>
> curl -u elastic:changeme [http://172.31.18.116:9200/filebeat\*/\_mapping/nginx\_dev\_access/field/geoip.ip?pretty](http://172.31.18.116:9200/filebeat*/_mapping/nginx_dev_access/field/geoip.ip?pretty)

Thanks for the reply, interestingly location is empty.

```
root@ip-172-31-18-116:~# curl -u elastic:changeme http://172.31.18.116:9200/filebeat-2017.06.30/_mapping/nginx.access/field/geoip.location?pretty
{ }
root@ip-172-31-18-116:~# curl -u elastic:changeme http://172.31.18.116:9200/filebeat-2017.06.30/_mapping/nginx.access/field/geoip.ip?pretty
{
  "filebeat-2017.06.30" : {
    "mappings" : {
      "nginx.access" : {
        "geoip.ip" : {
          "full_name" : "geoip.ip",
          "mapping" : {
            "ip" : {
              "type" : "keyword",
              "ignore_above" : 1024
            }
          }
        }
      }
    }
  }
}

```

However, these exist:

```
root@ip-172-31-18-116:~# curl -u elastic:changeme http://172.31.18.116:9200/filebeat-2017.06.30/_mapping/nginx.access/field/geoip.location.lon?pretty
{
  "filebeat-2017.06.30" : {
    "mappings" : {
      "nginx.access" : {
        "geoip.location.lon" : {
          "full_name" : "geoip.location.lon",
          "mapping" : {
            "lon" : {
              "type" : "float"
            }
          }
        }
      }
    }
  }
}
root@ip-172-31-18-116:~# curl -u elastic:changeme http://172.31.18.116:9200/filebeat-2017.06.30/_mapping/nginx.access/field/geoip.location.lat?pretty
{
  "filebeat-2017.06.30" : {
    "mappings" : {
      "nginx.access" : {
        "geoip.location.lat" : {
          "full_name" : "geoip.location.lat",
          "mapping" : {
            "lat" : {
              "type" : "float"
            }
          }
        }
      }
    }
  }
}

```

Note the \_mapping change is on purpose, we were tidying up some of the log inputs.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [June 30, 2017, 10:03am UTC](https://discuss.elastic.co/t/geoip-nginx-logs-no-drop-down-in-map-visual/91106/6 "2017-06-30T10:03:40Z")

</div>

What's your Logstash config look like, and your Elasticsearch mapping, for the entire index.

---

<div class="post-metadata">

**Author:** ![runtman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/runtman/32/18136_2.png) [@runtman](https://discuss.elastic.co/u/runtman)\
**Post date:** [June 30, 2017, 10:21am UTC](https://discuss.elastic.co/t/geoip-nginx-logs-no-drop-down-in-map-visual/91106/7 "2017-06-30T10:21:38Z")

</div>

Here is the logstash config:

```
filter {
        if [type] == "nginx.access" {
                grok {
                        match => { "message" => "%{HTTPD_COMBINEDLOG}" }
                }
                geoip {
                        source => "clientip"
                }
        }
        if [type] == "nginx.error" {
                grok {
                        match => { "message" => "%{HTTPD20_ERRORLOG}" }
                }
                geoip {
                        source => "clientip"
                }
        }
        if [type] == "nginx" {
                grok {
                        match => { "message" => "%{HTTPD_COMBINEDLOG}" }
                }
                geoip {
                        source => "clientip"
                }
        }
}

```

Ref the elastic search mapping, how should I send this wouldn't it be too much to paste?

I put it in pastebin, I hope that is ok:

[https://pastebin.com/uz6VL2bw](https://pastebin.com/uz6VL2bw)

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [June 30, 2017, 10:39am UTC](https://discuss.elastic.co/t/geoip-nginx-logs-no-drop-down-in-map-visual/91106/8 "2017-06-30T10:39:52Z")

</div>

Line 56-58 looks right. What does that look like in the index pattern settings in Kibana?

---

<div class="post-metadata">

**Author:** ![runtman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/runtman/32/18136_2.png) [@runtman](https://discuss.elastic.co/u/runtman)\
**Post date:** [June 30, 2017, 10:42am UTC](https://discuss.elastic.co/t/geoip-nginx-logs-no-drop-down-in-map-visual/91106/9 "2017-06-30T10:42:40Z")

</div>

![](https://us1.discourse-cdn.com/elastic/original/3X/9/1/91ad8fda7fa4dc473f40c199d9e9a0d10928cd9f.png) ![](https://us1.discourse-cdn.com/elastic/original/3X/9/8/9840caed1c1a6d88614fb3208079e83ca3153fa6.png)

Is this what you are after?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [June 30, 2017, 10:45am UTC](https://discuss.elastic.co/t/geoip-nginx-logs-no-drop-down-in-map-visual/91106/10 "2017-06-30T10:45:02Z")

</div>

Yep, but is there something for geoip.location?

---

<div class="post-metadata">

**Author:** ![runtman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/runtman/32/18136_2.png) [@runtman](https://discuss.elastic.co/u/runtman)\
**Post date:** [June 30, 2017, 10:46am UTC](https://discuss.elastic.co/t/geoip-nginx-logs-no-drop-down-in-map-visual/91106/11 "2017-06-30T10:46:12Z")

</div>

![](https://us1.discourse-cdn.com/elastic/original/3X/7/2/72de8d234bbb102a0a47761062a2e56c7900a02f.png)

Here is the search for that.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [June 30, 2017, 10:47am UTC](https://discuss.elastic.co/t/geoip-nginx-logs-no-drop-down-in-map-visual/91106/12 "2017-06-30T10:47:26Z")

</div>

Can you do a field refresh?  
Based on the Logstash config you posted I'd expect to see a `geoip.location`, but it's not there which is odd.

---

<div class="post-metadata">

**Author:** ![runtman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/runtman/32/18136_2.png) [@runtman](https://discuss.elastic.co/u/runtman)\
**Post date:** [June 30, 2017, 10:48am UTC](https://discuss.elastic.co/t/geoip-nginx-logs-no-drop-down-in-map-visual/91106/13 "2017-06-30T10:48:39Z")

</div>

Done, and no change.

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [June 30, 2017, 10:58am UTC](https://discuss.elastic.co/t/geoip-nginx-logs-no-drop-down-in-map-visual/91106/14 "2017-06-30T10:58:17Z")

</div>

Then there must be something else in your config working against this?

---

<div class="post-metadata">

**Author:** ![runtman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/runtman/32/18136_2.png) [@runtman](https://discuss.elastic.co/u/runtman)\
**Post date:** [June 30, 2017, 11:02am UTC](https://discuss.elastic.co/t/geoip-nginx-logs-no-drop-down-in-map-visual/91106/15 "2017-06-30T11:02:10Z")

</div>

Which configs?

Here is output of all logstash outputs

```
filter {
  if [type] == "syslog" {
    if [message] =~ /last message repeated [0-9]+ times/ {
      drop { }
    }
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => ["received_at", "%{@timestamp}"]
      add_field => ["received_from", "%{host}"]
    }
    syslog_pri { }
    date {
      match => ["syslog_timestamp", "MMM d HH:mm:ss", "MMM dd HH:mm:ss"]
    }
  }
}
filter {
        if [type] == "nginx.access" {
                grok {
                        match => { "message" => "%{HTTPD_COMBINEDLOG}" }
                }
                geoip {
                        source => "clientip"
                }
        }
        if [type] == "nginx.error" {
                grok {
                        match => { "message" => "%{HTTPD20_ERRORLOG}" }
                }
                geoip {
                        source => "clientip"
                }
        }
        if [type] == "nginx" {
                grok {
                        match => { "message" => "%{HTTPD_COMBINEDLOG}" }
                }
                geoip {
			source => "clientip"
		}
        }
}
filter {
  if [type] == "apache" {
      grok {
        match => { "message" => "%{COMBINEDAPACHELOG}"}
      }
      date {
        match => ["timestamp", "dd/MMM/yyyy:HH:mm:ss Z"]
      }
  }
}
filter {
  if [type] == "drupal" {
    grok {
      match => ["message", "%{SYSLOGTIMESTAMP} %{HOSTNAME} %{WORD}: %{URI:drupal_vhost}\|%{NUMBER:drupal_timestamp}\|(?<drupal_action>[^\|]*)\|%{IP:drupal_ip}\|(?<drupal_request_uri>[^\|]*)\|(?<drupal_referer>[^\|]*)\|(?<drupal_uid>[^\|]*)\|(?<drupal_link>[^\|]*)\|%{GREEDYDATA:drupal_message}" ]
    }
  }
}
output {
  elasticsearch {
    hosts => ["172.31.18.116:9200", "172.31.28.55:9200", "172.31.23.120:9200", "172.31.27.44:9200"]
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
    user => elastic
    password => changeme
  }
}
```

---

<div class="post-metadata">

**Author:** ![runtman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/runtman/32/18136_2.png) [@runtman](https://discuss.elastic.co/u/runtman)\
**Post date:** [July 3, 2017, 7:15am UTC](https://discuss.elastic.co/t/geoip-nginx-logs-no-drop-down-in-map-visual/91106/16 "2017-07-03T07:15:10Z")

</div>

I just deleted packet/metric indices, ran the templates to ES and started the indexing again and the map doesn't work for either of those two either. I installed the GEO plugin to ES only have I missed a step somewhere perhaps?

I have just done a fresh install of the entire stack, without packet/metric. Ensuring ES has the filebeat template before it has any data. Removing all filters other than ES output and nginx input and I still don't have any plots on the map.

I'm out of ideas ☹

---

<div class="post-metadata">

**Author:** ![runtman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/runtman/32/18136_2.png) [@runtman](https://discuss.elastic.co/u/runtman)\
**Post date:** [July 5, 2017, 7:31am UTC](https://discuss.elastic.co/t/geoip-nginx-logs-no-drop-down-in-map-visual/91106/17 "2017-07-05T07:31:29Z")

</div>

Can anyone assist with this, perhaps an Elastic member?

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [July 15, 2017, 10:29pm UTC](https://discuss.elastic.co/t/geoip-nginx-logs-no-drop-down-in-map-visual/91106/18 "2017-07-15T22:29:32Z")

</div>

Is this still not working for you?

---

<div class="post-metadata">

**Author:** ![runtman](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/runtman/32/18136_2.png) [@runtman](https://discuss.elastic.co/u/runtman)\
**Post date:** [July 16, 2017, 1:22pm UTC](https://discuss.elastic.co/t/geoip-nginx-logs-no-drop-down-in-map-visual/91106/19 "2017-07-16T13:22:41Z")

</div>

Nope, I have not been actively trouble shooting but this isn't working still.

---

<div class="post-metadata">

**Author:** ![josebonillajr](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/josebonillajr/32/20137_2.png) [@josebonillajr](https://discuss.elastic.co/u/josebonillajr)\
**Post date:** [July 17, 2017, 9:46pm UTC](https://discuss.elastic.co/t/geoip-nginx-logs-no-drop-down-in-map-visual/91106/20 "2017-07-17T21:46:40Z")

</div>

I am having the same issues however with iis logs. I have tried numerous ways to get my logs to show with geoip.location so I can visualize it. I am using version 5.5.

I have had the best luck using

```
   geoip {
      source=>"iis_client_ip" add_tag=>["geoip"]
    }

```

for my iis logging. But I am not able to visualize anything even though I can see location information.

[Next page](https://discuss.elastic.co/t/geoip-nginx-logs-no-drop-down-in-map-visual/91106.md?page=2)