GEOIP / Nginx logs, no drop down in map visual


(Anthony Cleaves) #1

Hello, so I want to visualise some of the Nginx logs we have with the geoip source.

So I have the following in Filebeat configurations to push the logs

- input_type: log
  paths:
    - /var/log/nginx/dev/access.log
  document_type: nginx_dev_access

Then in my nginx filter, I have the following GROK and filters:

filter {
        if [type] == "nginx_dev_access" {
                grok {
                        match => { "message" => "%{COMBINEDAPACHELOG}" }
                }
                geoip {
                        source => "clientip"
                        target => "geoip"
                        database => "/etc/logstash/GeoLiteCity.dat"
                        add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
                        add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}"  ]
                }
                mutate {
                        convert => [ "[geoip][coordinates]", "float"]
                }
        }
 }

If I go to the "Discover" home screen and do a search, I can see logs with these filters (image attached)

I see the available fields, they appear to be mark as unknown (the small ?)

So if I save this search and go to plot a map in visulations, I only have these options.

How do I get my custom filters to appear in that list?

It may be worth mentioning, when this ELK stack was built the default filebeats dashboards were adding thinking they would work, they do not as it goes through Logstash first.


(Anthony Cleaves) #2

Following other threads, I have reduced the filter to just this:

            geoip {
                    source => "clientip"
            }

I can again see data under geoip.ip but my drop down only has the 4 options, I'm assuming loading some template has meddled with that drop down box?

I have noticed the mapping is set to "keyword"

root@ip-172-31-25-192:/var/log# curl -u elastic:changeme http://172.31.18.116:9200/filebeat*/_mapping/nginx_dev_access/field/geoip.ip?pretty
{
  "filebeat-2017.06.28" : {
    "mappings" : {
      "nginx_dev_access" : {
        "geoip.ip" : {
          "full_name" : "geoip.ip",
          "mapping" : {
            "ip" : {
              "type" : "keyword",
              "ignore_above" : 1024
            }
          }
        }
      }
    }
  }
}

This needs to be geo_point I am guessing? Even if I empty logstash and elastic search ,load the default filebeat template. This data comes back as "keyword" is there anyway I can inforce it always be geo_point


(Anthony Cleaves) #3

Bump


(Mark Walkom) #4

What's the mapping for the geoip.location field, not the geoip.ip field.


(Anthony Cleaves) #5

Thanks for the reply, interestingly location is empty.

root@ip-172-31-18-116:~# curl -u elastic:changeme http://172.31.18.116:9200/filebeat-2017.06.30/_mapping/nginx.access/field/geoip.location?pretty
{ }
root@ip-172-31-18-116:~# curl -u elastic:changeme http://172.31.18.116:9200/filebeat-2017.06.30/_mapping/nginx.access/field/geoip.ip?pretty
{
  "filebeat-2017.06.30" : {
    "mappings" : {
      "nginx.access" : {
        "geoip.ip" : {
          "full_name" : "geoip.ip",
          "mapping" : {
            "ip" : {
              "type" : "keyword",
              "ignore_above" : 1024
            }
          }
        }
      }
    }
  }
}

However, these exist:

root@ip-172-31-18-116:~# curl -u elastic:changeme http://172.31.18.116:9200/filebeat-2017.06.30/_mapping/nginx.access/field/geoip.location.lon?pretty
{
  "filebeat-2017.06.30" : {
    "mappings" : {
      "nginx.access" : {
        "geoip.location.lon" : {
          "full_name" : "geoip.location.lon",
          "mapping" : {
            "lon" : {
              "type" : "float"
            }
          }
        }
      }
    }
  }
}
root@ip-172-31-18-116:~# curl -u elastic:changeme http://172.31.18.116:9200/filebeat-2017.06.30/_mapping/nginx.access/field/geoip.location.lat?pretty
{
  "filebeat-2017.06.30" : {
    "mappings" : {
      "nginx.access" : {
        "geoip.location.lat" : {
          "full_name" : "geoip.location.lat",
          "mapping" : {
            "lat" : {
              "type" : "float"
            }
          }
        }
      }
    }
  }
}

Note the _mapping change is on purpose, we were tidying up some of the log inputs.


(Mark Walkom) #6

What's your Logstash config look like, and your Elasticsearch mapping, for the entire index.


(Anthony Cleaves) #7

Here is the logstash config:

filter {
        if [type] == "nginx.access" {
                grok {
                        match => { "message" => "%{HTTPD_COMBINEDLOG}" }
                }
                geoip {
                        source => "clientip"
                }
        }
        if [type] == "nginx.error" {
                grok {
                        match => { "message" => "%{HTTPD20_ERRORLOG}" }
                }
                geoip {
                        source => "clientip"
                }
        }
        if [type] == "nginx" {
                grok {
                        match => { "message" => "%{HTTPD_COMBINEDLOG}" }
                }
                geoip {
                        source => "clientip"
                }
        }
}

Ref the elastic search mapping, how should I send this wouldn't it be too much to paste?

I put it in pastebin, I hope that is ok:

https://pastebin.com/uz6VL2bw


(Mark Walkom) #8

Line 56-58 looks right. What does that look like in the index pattern settings in Kibana?


(Anthony Cleaves) #9

Is this what you are after?


(Mark Walkom) #10

Yep, but is there something for geoip.location?


(Anthony Cleaves) #11

Here is the search for that.


(Mark Walkom) #12

Can you do a field refresh?
Based on the Logstash config you posted I'd expect to see a geoip.location, but it's not there which is odd.


(Anthony Cleaves) #13

Done, and no change.


(Mark Walkom) #14

Then there must be something else in your config working against this?


(Anthony Cleaves) #15

Which configs?

Here is output of all logstash outputs

filter {
  if [type] == "syslog" {
    if [message] =~ /last message repeated [0-9]+ times/ {
      drop { }
    }
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    syslog_pri { }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
}
filter {
        if [type] == "nginx.access" {
                grok {
                        match => { "message" => "%{HTTPD_COMBINEDLOG}" }
                }
                geoip {
                        source => "clientip"
                }
        }
        if [type] == "nginx.error" {
                grok {
                        match => { "message" => "%{HTTPD20_ERRORLOG}" }
                }
                geoip {
                        source => "clientip"
                }
        }
        if [type] == "nginx" {
                grok {
                        match => { "message" => "%{HTTPD_COMBINEDLOG}" }
                }
                geoip {
			source => "clientip"
		}
        }
}
filter {
  if [type] == "apache" {
      grok {
        match => { "message" => "%{COMBINEDAPACHELOG}"}
      }
      date {
        match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
      }
  }
}
filter {
  if [type] == "drupal" {
    grok {
      match => ["message", "%{SYSLOGTIMESTAMP} %{HOSTNAME} %{WORD}: %{URI:drupal_vhost}\|%{NUMBER:drupal_timestamp}\|(?<drupal_action>[^\|]*)\|%{IP:drupal_ip}\|(?<drupal_request_uri>[^\|]*)\|(?<drupal_referer>[^\|]*)\|(?<drupal_uid>[^\|]*)\|(?<drupal_link>[^\|]*)\|%{GREEDYDATA:drupal_message}" ]
    }
  }
}
output {
  elasticsearch {
    hosts => ["172.31.18.116:9200", "172.31.28.55:9200", "172.31.23.120:9200", "172.31.27.44:9200"]
    index => "%{[@metadata][beat]}-%{+YYYY.MM.dd}"
    document_type => "%{[@metadata][type]}"
    user => elastic
    password => changeme
  }
}

(Anthony Cleaves) #16

I just deleted packet/metric indices, ran the templates to ES and started the indexing again and the map doesn't work for either of those two either. I installed the GEO plugin to ES only have I missed a step somewhere perhaps?

I have just done a fresh install of the entire stack, without packet/metric. Ensuring ES has the filebeat template before it has any data. Removing all filters other than ES output and nginx input and I still don't have any plots on the map.

I'm out of ideas :frowning:


(Anthony Cleaves) #17

Can anyone assist with this, perhaps an Elastic member?


(Mark Walkom) #18

Is this still not working for you?


(Anthony Cleaves) #19

Nope, I have not been actively trouble shooting but this isn't working still.


(Jose Bonilla) #20

I am having the same issues however with iis logs. I have tried numerous ways to get my logs to show with geoip.location so I can visualize it. I am using version 5.5.

I have had the best luck using

   geoip {
      source=>"iis_client_ip" add_tag=>["geoip"]
    }

for my iis logging. But I am not able to visualize anything even though I can see location information.