I'm unable to create "Maps" visualization. I get the error "Selected index pattern does not contain source and destination fields." I'm on version 7.13.2 ELK
I used geoip to define the source and destination IP fields as below :-
if "BackendIP" not in [tags]
{
geoip
{
target => "Src_IP"
source => "Fwd_Client_IP"
tag_on_failure => ["IP-lookup-failed"]
}
geoip
{
target => "Dst_IP"
source => "Forward_IP"
tag_on_failure => ["IP-lookup-failed"]
}
I read some articles about creating a geo_point , but since geoip already creates "lon" and "lat" values, doesn't already have what it needs to create a geo_point? If not, could you help me how to create a geo_point ? can it created on logstash config ? or elasticsearch ?
i'm new to ELK and in a learning phase .
What is the mapping output for the index. The lat and lon fields must be mapped as geo_point in order to be used in the maps application. If they are just mapped as numbers then you do not have a spatial index and can not use the data in maps.
Hmm, seems the mapping is missing from my index . How do i create the geo_point ? is it done on logstash? via config ?
After index creation and before inserting documents, add geo_point mapping to your index by running a command like the one below in Kibana => dev tools => console.
PUT your_index_name/_mapping
{
"properties": {
"location": {
"type": "geo_point"
}
}
}
awesome 
I was able to create . But i still get same error "Selected index pattern does not contain source and destination fields." , seems i'm missing something .
"timezone": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}
}
},
"line_type": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
},
"location": {
"type": "geo_point"
}
}
}
}
}```
i changed "location" to "geolocation" and it worked , the error has disappeared. However it shows that no results , even after ingesting some logs . Does changing from location to geolocation have some significance ?
PUT logstash_geo_ghost1/_mapping
{
"properties":{
"geolocation": {
"type":"geo_point"
}
}
any update ? No data is shown even after processing some logs
logstash parses correctly , source and destination IPs are visible
{
"Src_IP" => {
"country_code3" => "FR",
"postal_code" => "13000",
"location" => {
"lat" => 43.2951,
"lon" => 5.3861
},
"ip" => "2.21.85.4",
"country_code2" => "FR",
"continent_code" => "EU",
"latitude" => 43.2951,
"region_name" => "Bouches-du-Rhône",
"country_name" => "France",
"city_name" => "Marseille",
"region_code" => "13",
"longitude" => 5.3861,
"timezone" => "Europe/Paris"
},
"Dst_IP" => {
"country_code3" => "IN",
"postal_code" => "600001",
"location" => {
"lat" => 12.8996,
"lon" => 80.2209
},
"ip" => "23.57.75.218",
"country_code2" => "IN",
"continent_code" => "AS",
"latitude" => 12.8996,
"region_name" => "Tamil Nadu",
"country_name" => "India",
"city_name" => "Chennai",
"region_code" => "TN",
"longitude" => 80.2209,
"timezone" => "Asia/Kolkata"
Output section :
output
{
elasticsearch
{
hosts => [ "172.27.205.251:9200" ]
index => "logstash_geo"
}
stdout{}
}
@Shreesh_Narayanan
The source-and-destination selection in those two dropdowns should both be a different field of type geo_point. It looks like you are using the same field location.
Also a few other things to check:
- make sure that the data is indexed correctly (e.g. use Discover to check you have two
geo_point fields with actual lat/lons,
- make sure the time-filter (top right, in Kibana) is large enough for your data-range
oh okay, makes sense . To create geo_point for source and destination, i need to have the lon/lat values in a single field,correct ? how do i create it ? i tried adding a field inside geoip such as the one below , but it did not work .should i use a separate mutate filter for this ?
geoip
{
target => "Dst_IP"
source => "Forward_IP"
tag_on_failure => ["IP-lookup-failed"]
add_field => {"Destination_geo" => "[geoip][location][lon]","[geoip]location][lat]"]
}
I managed to get the source and destination lat/lon points in the same field
"destinationip" => "42.106.161.193",
"sourceip" => "1.1.1.1",
"sourcelocation" => "143.2104,-33.494",
"destlocation" => "88.3832,22.518",
How should the geo_point be created now ? i think it should be for source and destination .
PUT logstashgeotest/_mapping
{
"properties":{
"sourcelocation":{
"type": "geo_point"
}
}
}
I tried the above, but elasticsearch (done via dev console) gave an error
{
"error" : {
"root_cause" : [
{
"type" : "illegal_argument_exception",
"reason" : "mapper [sourcelocation] cannot be changed from type [text] to [geo_point]"
}
],
"type" : "illegal_argument_exception",
"reason" : "mapper [sourcelocation] cannot be changed from type [text] to [geo_point]"
},
"status" : 400
}
You can not change a field's mapping. You will need to delete the index and then use the correct mappings.
thank you . it works now .
sample config where lat and lon values are in the same field and converted to geo point , this conversion was done before any data was pushed/ingested into it
geoip
{
add_tag => [ "geoip" ]
source => "sourceip"
target => "SRC_ip"
#fields => [ "ip", "country_code2", "country_name", "latitude", "longitude" ]
}
geoip
{
add_tag => ["geoip"]
source => "destinationip"
target => "Dest_IP"
}
mutate
{
add_field => ["sourcelocation","%{[SRC_ip][longitude]}","tmplat","%{[SRC_ip][latitude]}"]
add_field => ["destlocation","%{[Dest_IP][longitude]}","tmplatdst","%{[Dest_IP][latitude]}"]
}
mutate
{
merge => ["sourcelocation","tmplat"]
merge => ["destlocation","tmplatdst"]
}
mutate
{
convert => [ "sourcelocation","float"]
convert => ["destlocation","float"]
}
mutate {
remove_field => [ "tmplat","tmplatdst"]
}
