GEOIP / Nginx logs, no drop down in map visual

Elastic Stack Kibana
1

Hello, so I want to visualise some of the Nginx logs we have with the geoip source.

So I have the following in Filebeat configurations to push the logs

- input_type: log
  paths:
    - /var/log/nginx/dev/access.log
  document_type: nginx_dev_access

Then in my nginx filter, I have the following GROK and filters:

filter {
        if [type] == "nginx_dev_access" {
                grok {
                        match => { "message" => "%{COMBINEDAPACHELOG}" }
                }
                geoip {
                        source => "clientip"
                        target => "geoip"
                        database => "/etc/logstash/GeoLiteCity.dat"
                        add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
                        add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}"  ]
                }
                mutate {
                        convert => [ "[geoip][coordinates]", "float"]
                }
        }
 }

If I go to the "Discover" home screen and do a search, I can see logs with these filters (image attached)

Screenshot from 2017-06-28 13-55-42.png1267×392 32.7 KB

I see the available fields, they appear to be mark as unknown (the small ?)

So if I save this search and go to plot a map in visulations, I only have these options.

Selection_002.png442×647 43.7 KB

How do I get my custom filters to appear in that list?

It may be worth mentioning, when this ELK stack was built the default filebeats dashboards were adding thinking they would work, they do not as it goes through Logstash first.

2

Following other threads, I have reduced the filter to just this:

            geoip {
                    source => "clientip"
            }

I can again see data under geoip.ip but my drop down only has the 4 options, I'm assuming loading some template has meddled with that drop down box?

I have noticed the mapping is set to "keyword"

root@ip-172-31-25-192:/var/log# curl -u elastic:changeme http://172.31.18.116:9200/filebeat*/_mapping/nginx_dev_access/field/geoip.ip?pretty
{
  "filebeat-2017.06.28" : {
    "mappings" : {
      "nginx_dev_access" : {
        "geoip.ip" : {
          "full_name" : "geoip.ip",
          "mapping" : {
            "ip" : {
              "type" : "keyword",
              "ignore_above" : 1024
            }
          }
        }
      }
    }
  }
}

This needs to be geo_point I am guessing? Even if I empty logstash and elastic search ,load the default filebeat template. This data comes back as "keyword" is there anyway I can inforce it always be geo_point

3

Bump

4

What's the mapping for the geoip.location field, not the geoip.ip field.

5

Thanks for the reply, interestingly location is empty.

root@ip-172-31-18-116:~# curl -u elastic:changeme http://172.31.18.116:9200/filebeat-2017.06.30/_mapping/nginx.access/field/geoip.location?pretty
{ }
root@ip-172-31-18-116:~# curl -u elastic:changeme http://172.31.18.116:9200/filebeat-2017.06.30/_mapping/nginx.access/field/geoip.ip?pretty
{
  "filebeat-2017.06.30" : {
    "mappings" : {
      "nginx.access" : {
        "geoip.ip" : {
          "full_name" : "geoip.ip",
          "mapping" : {
            "ip" : {
              "type" : "keyword",
              "ignore_above" : 1024
            }
          }
        }
      }
    }
  }
}

However, these exist:

root@ip-172-31-18-116:~# curl -u elastic:changeme http://172.31.18.116:9200/filebeat-2017.06.30/_mapping/nginx.access/field/geoip.location.lon?pretty
{
  "filebeat-2017.06.30" : {
    "mappings" : {
      "nginx.access" : {
        "geoip.location.lon" : {
          "full_name" : "geoip.location.lon",
          "mapping" : {
            "lon" : {
              "type" : "float"
            }
          }
        }
      }
    }
  }
}
root@ip-172-31-18-116:~# curl -u elastic:changeme http://172.31.18.116:9200/filebeat-2017.06.30/_mapping/nginx.access/field/geoip.location.lat?pretty
{
  "filebeat-2017.06.30" : {
    "mappings" : {
      "nginx.access" : {
        "geoip.location.lat" : {
          "full_name" : "geoip.location.lat",
          "mapping" : {
            "lat" : {
              "type" : "float"
            }
          }
        }
      }
    }
  }
}

Note the _mapping change is on purpose, we were tidying up some of the log inputs.

6

What's your Logstash config look like, and your Elasticsearch mapping, for the entire index.

7

Here is the logstash config:

filter {
        if [type] == "nginx.access" {
                grok {
                        match => { "message" => "%{HTTPD_COMBINEDLOG}" }
                }
                geoip {
                        source => "clientip"
                }
        }
        if [type] == "nginx.error" {
                grok {
                        match => { "message" => "%{HTTPD20_ERRORLOG}" }
                }
                geoip {
                        source => "clientip"
                }
        }
        if [type] == "nginx" {
                grok {
                        match => { "message" => "%{HTTPD_COMBINEDLOG}" }
                }
                geoip {
                        source => "clientip"
                }
        }
}

Ref the elastic search mapping, how should I send this wouldn't it be too much to paste?

I put it in pastebin, I hope that is ok:

https://pastebin.com/uz6VL2bw

8

Line 56-58 looks right. What does that look like in the index pattern settings in Kibana?

9

Screenshot from 2017-06-30 11-42-13.png1424×236 18.9 KB
Screenshot from 2017-06-30 11-42-02.png1428×856 70.4 KB

Is this what you are after?

10

Yep, but is there something for geoip.location?

11

Screenshot from 2017-06-30 11-45-48.png1433×289 24.5 KB

Here is the search for that.

12

Can you do a field refresh?
Based on the Logstash config you posted I'd expect to see a geoip.location, but it's not there which is odd.

13

Done, and no change.

14

Then there must be something else in your config working against this?

15

Which configs?

Here is output of all logstash outputs

filter {
  if [type] == "syslog" {
    if [message] =~ /last message repeated [0-9]+ times/ {
      drop { }
    }
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    syslog_pri { }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
}
filter {
        if [type] == "nginx.access" {
                grok {
                        match => { "message" => "%{HTTPD_COMBINEDLOG}" }
                }
                geoip {
                        source => "clientip"
                }
        }
        if [type] == "nginx.error" {
                grok {
                        match => { "message" => "%{HTTPD20_ERRORLOG}" }
                }
                geoip {
                        source => "clientip"
                }
        }
        if [type] == "nginx" {
                grok {
                        match => { "message" => "%{HTTPD_COMBINEDLOG}" }
                }
                geoip {
			source => "clientip"
		}
        }
}
filter {
  if [type] == "apache" {
      grok {
        match => { "message" => "%{COMBINEDAPACHELOG}"}
      }
      date {
        match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
      }
  }
}
filter {
  if [type] == "drupal" {
    grok {
      match => ["message", "%{SYSLOGTIMESTAMP} %{HOSTNAME} %{WORD}: %{URI:drupal_vhost}\|%{NUMBER:drupal_timestamp}\|(?<drupal_action>[^\|]*)\|%{IP:drupal_ip}\|(?<drupal_request_uri>[^\|]*)\|(?<drupal_referer>[^\|]*)\|(?<drupal_uid>[^\|]*)\|(?<drupal_link>[^\|]*)\|%{GREEDYDATA:drupal_message}" ]
    }
  }
}
output {
  elasticsearch {
    hosts => ["172.31.18.116:9200", "172.31.28.55:9200", "172.31.23.120:9200", "172.31.27.44:9200"]
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
    user => elastic
    password => changeme
  }
}
16

I just deleted packet/metric indices, ran the templates to ES and started the indexing again and the map doesn't work for either of those two either. I installed the GEO plugin to ES only have I missed a step somewhere perhaps?

I have just done a fresh install of the entire stack, without packet/metric. Ensuring ES has the filebeat template before it has any data. Removing all filters other than ES output and nginx input and I still don't have any plots on the map.

I'm out of ideas :frowning:

17

Can anyone assist with this, perhaps an Elastic member?

18

Is this still not working for you?

19

Nope, I have not been actively trouble shooting but this isn't working still.

20

I am having the same issues however with iis logs. I have tried numerous ways to get my logs to show with geoip.location so I can visualize it. I am using version 5.5.

I have had the best luck using

   geoip {
      source=>"iis_client_ip" add_tag=>["geoip"]
    }

for my iis logging. But I am not able to visualize anything even though I can see location information.