Not able to plot geo data on Kibana

Hi Guys,

So I have an issue with displaying the geolocation data on Kibana map visualization. Sorry if my description becomes too long but I will try to provide as much information as possible.

Past:

I am using logstash to ingest data, parse/transform, and send over to elastic search. I had logs coming in from WAF but everything within the message field. Converted that into JSON format to perform further operations. One of those fields was IP field. I activated geoip option for the filter plugin in logstash assuming I might be able to plot the country on a map based on source IP.

Present:

Logs are coming in as they should and I am able to create visualizations using other data points. The only thing I am not able to do is map the locations of source IP on kibana map. When I try to add a layer on kibana map using the 'Document' option, I do not see the desired index in the drop-down.

I did a bit of searching and found that in order for my index to be visible as a valid option I need to have a geo_point data type in my index. I looked at the available data fields in my index related to geo location and found multiple fields and figured out that I would need geoip.location field as geo-point data type. The issue is I have no idea as to how to do that. I looked at the mapping of the fields in index setting page and found that the fields are mapped to float (I have provided information below).

I also looked at logstash documentation for geoip option and it states that mapping is automatically done to geo_data point when using elasticsearch output plugin. But i don't see the same.

Logstash Setting:

filter {

        json {
                source => "message"

}

        ruby {
                path => "/etc/logstash/rubyScripts/flattenJSON.rb"
                script_params => { "field" => "httpRequest" }
}

        ruby {
                path => "/etc/logstash/rubyScripts/headersFaltten.rb"
                script_params => { "field" => "httpRequest.headers" }
}

        geoip {
                source => "httpRequest.clientIp"
}

        date {
                match => ["timestamp", "UNIX_MS"]
                target => "@timestamp"
}

        mutate {
                remove_field => ["message", "timestamp", "httpRequest.country"]
}

}

GeoIP fields available in my index:

geoip.city_name, geoip.continent_code, geoip.country_code2, geoip.country_code3, geoip.country_name, geoip.dma_code, geoip.ip, geoip.latitude, geoip.location.lat, geoip.location.lon, geoip.longitude, geoip.postal_code, geoip.region_code, geoip.region_name, geoip.timezone

Default Geoip mapping setting in my index:

"geoip": {
          "properties": {
            "city_name": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "continent_code": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "country_code2": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "country_code3": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "country_name": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "dma_code": {
              "type": "long"
            },
            "ip": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "latitude": {
              "type": "float"
            },
            "location": {
              "properties": {
                "lat": {
                  "type": "float"
                },
                "lon": {
                  "type": "float"
                }
              }
            },
            "longitude": {
              "type": "float"
            },
            "postal_code": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "region_code": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "region_name": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            },
            "timezone": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            }
          }
        }

From logstash documentation online:

"A [geoip][location] field is created if the GeoIP lookup returns latitude and longitude. The field is stored in the GeoJSON format. Additionally, the default Elasticsearch template provided with the elasticsearch output maps the [geoip][location] field to an Elasticsearch Geo_point datatype."

Please help me with this issue. And also let me know if you need any more information.

Kibana version: 7.7
Experience with ELK: 3-4 months

Welcome to our community! :smiley: We aren't all guys though.

Your location is not mapped as a geo_point which is the problem. What does an example document look like?

Hi @warkolm, Thank you for the reply, and really sorry for replying a bit late. Below is a sample document. It is basically AWS WAF logs pulled from S3 bucket. I have removed sensitive content from the original log line.

{
   "timestamp":1600214733254,
   "formatVersion":1,
   "webaclId":"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
   "terminatingRuleId":"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
   "terminatingRuleType":"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
   "action":"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
   "terminatingRuleMatchDetails":[
      
   ],
   "httpSourceName":"ALB",
   "httpSourceId":"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
   "ruleGroupList":[
      {
         "ruleGroupId":"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
         "terminatingRule":null,
         "nonTerminatingMatchingRules":[
            
         ],
         "excludedRules":null
      },
      {
         "ruleGroupId":"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
         "terminatingRule":null,
         "nonTerminatingMatchingRules":[
            
         ],
         "excludedRules":null
      }
   ],
   "rateBasedRuleList":[
      
   ],
   "nonTerminatingMatchingRules":[
      
   ],
   "httpRequest":{
      "clientIp":"196.52.43.125",
      "country":"ZA",
      "headers":[
         {
            "name":"Host",
            "value":"XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
         },
         {
            "name":"User-Agent",
            "value":"Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3602.2 Safari/537.36"
         },
         {
            "name":"Accept",
            "value":"*/*"
         }
      ],
      "uri":"/",
      "args":"",
      "httpVersion":"HTTP/1.0",
      "httpMethod":"GET",
      "requestId":null
   }
}

So this document is pulled in by logstash, which comes within the field message. I then convert the message into JSON structure using JSON filter plugin. After that I try to flatten the nested fields among which one field is clientIP. Then I run a geoip filter plugin on this field to get the related data points mentioned in my actual post.

And also, a few weeks back we had issues with open shards running to limits. So in order to remediate that we had to create index templates and policies. I believe this newly defined index template overwrote the default template and now when I am looking at the template settings I see the mapping to be empty. Will I have to mark geoip.location as geo_point data type in this setting from Kibana?

The mapping tab for the newly created index template shows the following:

{
  "_doc": {
    "_meta": {},
    "_source": {},
    "properties": {}
  }
}