Geo_ip field not showing on tile map


(Theo Delahayes) #1

Hello, and sorry in advance if i didn't put the topic in the right subject.
My problem is that i can't get the geo_point to show in Kibana.
Here is my Logstash conf :

input {  
  beats {
    # The port to listen on for filebeat connections.
    port => 5044
    # The IP address to listen for filebeat connections.
    host => "0.0.0.0"
    client_inactivity_timeout => 160
  }
}
filter {
  if [fileset][module] == "nginx" {
    if [fileset][name] == "access" {
      grok {
        match => { "message" => ["%{IPORHOST:[nginx][access][remote_ip]} - %{DATA:[nginx][access][user_name]} \[%{HTTPDATE:[nginx][access][time]}\] \"%{WORD:[nginx][access][method]} %{DATA:[nginx][access][url]} HTTP/%{NUMBER:[nginx][access][http_version]}\" %{NUMBER:[nginx][access][response_code]} %{NUMBER:[nginx][access][body_sent][bytes]} \"%{DATA:[nginx][access][referrer]}\" \"%{DATA:[nginx][access][agent]}\""] }
        remove_field => "message"
      }
      mutate {
        add_field => { "read_timestamp" => "%{@timestamp}" }
      }
      date {
        match => [ "[nginx][access][time]", "dd/MMM/YYYY:H:m:s Z" ]
        remove_field => "[nginx][access][time]"
      }
      useragent {
        source => "[nginx][access][agent]"
        target => "[nginx][access][user_agent]"
        remove_field => "[nginx][access][agent]"
      }
      geoip {
        source => "[nginx][access][remote_ip]"
        target => "[nginx][access][geoip]"
      }
    }
    else if [fileset][name] == "error" {
      grok {
        match => { "message" => ["%{DATA:[nginx][error][time]} \[%{DATA:[nginx][error][level]}\] %{NUMBER:[nginx][error][pid]}#%{NUMBER:[nginx][error][tid]}: (\*%{NUMBER:[nginx][error][connection_id]} )?%{GREEDYDATA:[nginx][error][message]}"] }
        remove_field => "message"
      }
      mutate {
        rename => { "@timestamp" => "read_timestamp" }
      }
      date {
        match => [ "[nginx][error][time]", "YYYY/MM/dd H:m:s" ]
        remove_field => "[nginx][error][time]"
      }
    }
  }
}
output {
    elasticsearch {
        hosts => "localhost"
        index => "logstash-nginx-%{+YYYY.MM.dd}"
    }
}

It seems like the mapping is done correctly at least the mapping of one of the indexes generated lets it believe :

  "geoip": {
    "dynamic": "true",
    "properties": {
      "ip": {
        "type": "ip"
      },
      "latitude": {
        "type": "half_float"
      },
      "location": {
        "type": "geo_point"
      },
      "longitude": {
        "type": "half_float"
      }
    }
  } 

And when i search for the said documents it appears that no error have occurred :

"geoip": {
            "ip": "195.218.14.146",
            "continent_code": "EU",
            "country_code2": "LU",
            "region_name": "Esch-sur-Alzette",
            "country_name": "Luxembourg",
            "location": {
              "lat": 49.4958,
              "lon": 5.9806
            },
            "city_name": "Esch-sur-Alzette",
            "longitude": 5.9806,
            "country_code3": "LU",
            "timezone": "Europe/Luxembourg",
            "latitude": 49.4958,
            "region_code": "ES"
          }

Yet when i try to visualize data nothing is showing, and i checked more than once if the requests were within the dates at the top right of the screen.
I'm completely lost ..

Thanks in advance


(Bill McConaghy) #2

Can you share your visualization configuration? Everything you posted looks correct to me.


(Theo Delahayes) #3

Hi, thanks for replying so fast :smiley:

Well i did not have many choices in the field i was choosing but here they are :

The time filter is set to today as i curled the nginx couples of time not so long ago.
And nothing is showing on the map


(Bill McConaghy) #4

The config all looks correct. Do you see any errors in the browser or in Kibana logs?


(Theo Delahayes) #5

I only have "code 200" in logs, and when Kibana searches for the data to populate the map it finds the right index but however there isn't any hits :

    {
    "responses": [
        {
            "took": 1,
            "timed_out": false,
            "_shards": {
                "total": 5,
                "successful": 5,
                "skipped": 0,
                "failed": 0
            },
            "hits": {
                "total": 15,
                "max_score": 0,
                "hits": []
            },
            "aggregations": {
                "filter_agg": {
                    "2": {
                        "buckets": []
                    },
                    "doc_count": 0
                }
            },
            "status": 200
        }
    ]
}

(Bill McConaghy) #6

What's the search it is doing look like?


(Theo Delahayes) #7

So i got this by copying the request of the search in cURL format and i rearranged it because it was pretty bulky.
This is what was under the --data-binary :
Is the "size":0 normal ?

{
  "index":["logstash-nginx-2018.06.18"],
  "ignore_unavailable":true,
  "preference":1529332880274
}
{
  "size":0,
  "_source":
  {
    "excludes":[]
  },
  "aggs":
  {
    "filter_agg":
    {
      "filter":
      {
        "geo_bounding_box":
        {
          "geoip.location":
          {
            "top_left":
            {
              "lat":90,
              "lon":-180
            },
            "bottom_right":
            {
              "lat":-90,
              "lon":180
            }
          }
        }
      },
      "aggs":
      {
        "2":
        {
          "geohash_grid":
          {
            "field": "geoip.location",
            "precision":2
          },
          "aggs":
          {
            "3":
            {
              "geo_centroid":
              {
                "field":"geoip.location"
              }
            }
          }
        }
      }
    }
  },
  "stored_fields":["*"],
  "script_fields":{},
  "docvalue_fields":["@timestamp","read_timestamp"],
  "query":
  {
    "bool":
    {
      "must":[
      {
        "match_all":{}
      },
      {
        "range":
        {
          "@timestamp": 
          {
            "gte":1529272800000,
            "lte":1529359199999,
            "format":"epoch_millis"
          }
        }
      }],
      "filter":[],
      "should":[],
      "must_not":[]
    }
  }
}

and this is the rest of the request :

curl 'https://my.kibana/elasticsearch/_msearch' \
-XPOST \
-H 'Content-Type: application/x-ndjson' \
-H 'Origin: https://my.kibana' \
-H 'Accept: application/json, text/plain, */*' \
-H 'User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_6) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/11.1 Safari/605.1.15' \
-H 'Referer: https://my.kibana/app/kibana' \
-H 'kbn-version: 6.2.4' \

(Bill McConaghy) #8

that size 0 looks suspicious to me for sure.


(Theo Delahayes) #9

I found it !

Thanks to getting to the request that Kibana was sending I understood where the problem came from : the geoip field must be "at level 0" if that makes any sense, for me it was under "nginx.access.geoip.location", where it has to be under "geoip.location".
Kibana was therefore unable to find the field.

Also i learned that the "size": 0 was normal, it's apparently just to prevent hits to show up when we use aggregations in order to get just the result of the aggregations.

Thanks a lot for helping solve this problem and for your admirable response speed :smiley:


(system) #10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.