Geo_point coming from geoip are not plotting in Kibana

Elastic Stack Kibana
1

I have been tweaking my mappings and logstash pipeline configurations for days now trying to figure out hwo to plot GeoIPs to no avail. I have no idea what to do anymore.

I have a template:

{
  "template" : "httpd*",
  "version" : 1,
  "settings" : {
    "index.refresh_interval" : "5s",
    "number_of_replicas": 0
  },
  "mappings": {
    "apache_logs": {
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "@version": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "geoip": {
          "properties": {
            ...
            "latitude": {
              "type": "float"
            },
            "location": {
              "type": "geo_point"
            },
            "longitude": {
              "type": "float"
            },
           ...
          }
        },
      ... # cut out for char limit
      }
    }
  }
}

I use this logstash configuration:

input {
  beats {
    port => 5044
    tags => "apache"
  }
}

filter {

  if "apache" in [tags] {
    grok {
      match => { "message" => ["%{COMBINEDAPACHELOG}", ........ ] }
      remove_field => "message"
      tag_on_failure => ["no_apache_match"]
    }
    if "no_apache_match" not in [tags] {
      geoip {
        source => "clientip"
        target => "geoip"
        add_field => [ "[geoip][coordinates]", "%{[geoip][longitude]}" ]
        add_field => [ "[geoip][coordinates]", "%{[geoip][latitude]}"  ]
      }
    }
    mutate {
      convert => {
          "[geoip][coordinates]" => "float"
      }
    }
    ...
  }
  
}

output {

  if "apache" in [tags] {
    if "apache_fiesta" in [tags] {
      elasticsearch {
        hosts => ["localhost:9200"]
        user => "elastic"
        password => "MrR0bot$"
        index => "httpd_fiesta-%{+YYYY.MM.dd}"
        document_type => "apache_logs"
      }
    }
  }
  ...
}

I check that the mapping is being applied:

# curl -u elastic GET 'http://localhost:9200/httpd_fiesta-2017.10.09/_mapping'
{
  "httpd_fiesta-2017.10.09": {
    "mappings": {
      "apache_logs": {
        "properties": {
         .....
          "geoip": {
            "properties": {
               ....
              "latitude": {
                "type": "float"
              },
              "location": {
                "type": "geo_point"
              },
              "longitude": {
                "type": "float"
              },
             ...
            }
          },
          ...
        }
      }
    }
  }
}

So geoip.location is indeed a geo_point data type.

In Kibana, I have refreshed the field list for my index pattern.

I have deleted this index prior to any recent changes, so all of my log entries are uniform.

In the Discover page, I have documents that look like this:

{
  "_index": "httpd_fiesta-2017.10.09",
  "_type": "apache_logs",
  "_source": {
    ....
    "geoip": {
       ...
      "location": {
        "lon": -106.7177,
        "lat": 35.3275
      },
      ...
    },
   ...
    "tags": [
      "apache_fiesta",
      "apache",
      "beats_input_codec_plain_applied"
    ],
     ...
  },
 ....
}

As you can see, there are no geoip failures, and the data is collected correctly.

Kibana shows a globe symbol next to geoip.location, and when i click it, it gives me the option to visualize.

On the visualize tab, I can select Aggregation->Geohash and Field->geoip.location just fine.

However, there is no data! I am using a valid time frame for which there is data in the discover tab.

The centroid label in the bottom right corner says NaN- NaN, which tells me it is not aggregating anything.

What the heck is going on?

2

Hmm. I've just discovered that I can change Options->Map type in the tile map visualization to any of the following:

  • Shaded circle markers
  • Shaded Geohash Grid
  • Heatmap

and it actually works!

But if I try to use Scaled Circle Markers, nothing shows up, no matter how far I zoom in!!

This has to be a bug unless I've missed documentation on how the scaled markers work.

3

What version are you on?

Also this is a pointless config option;

See GeoIP in the Elastic Stack - Elasticsearch, Logstash, Ingest API | Elastic Blog

4

Yes I realize I'm not actually using that new field for anything. It is a relic of my debugging phase and I left it in just in case of conflicts. I believe I found that on some question here or some blog.
I am using 5.6 across the board.

I pretty much went through everything in that link. To be clear, I discovered I am seeing plotted data on my map.

But it only works if I choose something other than scaled circle markers. That seems strange and inconsistent.

I'm going to give your blog post a thorough read through to see if anything stands out.

UPDATE:

Soooo, I just discovered something interesting.

If I save the visualization and come back to it, the scaled circle marker option works!

You cannot go straight from the discover page using the visualize shortcut under the field that is a geo_point type (in this case geoip.location).

This has to be a bug ???

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.