Unable to create "Maps" visualization - Getting "Selected index pattern does not contain source and destination fields."

[Shreesh_Narayanan]

I'm unable to create "Maps" visualization. I get the error "Selected index pattern does not contain source and destination fields." I'm on version 7.13.2 ELK

I used geoip to define the source and destination IP fields as below :-

 if "BackendIP" not in [tags]
                {
        geoip
                {
                        target => "Src_IP"
                        source => "Fwd_Client_IP"
                        tag_on_failure => ["IP-lookup-failed"]

                }
        geoip
                {
                        target => "Dst_IP"
                        source => "Forward_IP"
                        tag_on_failure => ["IP-lookup-failed"]
                }

I read some articles about creating a geo_point , but since geoip already creates "lon" and "lat" values, doesn't already have what it needs to create a geo_point? If not, could you help me how to create a geo_point ? can it created on logstash config ? or elasticsearch ?

i'm new to ELK and in a learning phase .

[Nathan_Reese]

What is the mapping output for the index. The lat and lon fields must be mapped as geo_point in order to be used in the maps application. If they are just mapped as numbers then you do not have a spatial index and can not use the data in maps.

[Shreesh_Narayanan]

Hmm, seems the mapping is missing from my index . How do i create the geo_point ? is it done on logstash? via config ?

[Nathan_Reese]

After index creation and before inserting documents, add geo_point mapping to your index by running a command like the one below in Kibana => dev tools => console.

PUT your_index_name/_mapping
{
  "properties": {
    "location": {
      "type": "geo_point"
    }
  }
}

[Shreesh_Narayanan]

awesome I was able to create . But i still get same error "Selected index pattern does not contain source and destination fields." , seems i'm missing something .

"timezone": {
              "type": "text",
              "fields": {
                "keyword": {
                  "type": "keyword",
                  "ignore_above": 256
                }
              }
            }
          }
        },
        "line_type": {
          "type": "text",
          "fields": {
            "keyword": {
              "type": "keyword",
              "ignore_above": 256
            }
          }
        },
        "location": {
          "type": "geo_point"
        }
      }
    }
  }
}```

[Shreesh_Narayanan]

i changed "location" to "geolocation" and it worked , the error has disappeared. However it shows that no results , even after ingesting some logs . Does changing from location to geolocation have some significance ?

PUT logstash_geo_ghost1/_mapping
{
    "properties":{
      "geolocation": {
        "type":"geo_point"
      }
    }

[Shreesh_Narayanan]

any update ? No data is shown even after processing some logs

logstash parses correctly , source and destination IPs are visible

{
                     "Src_IP" => {
         "country_code3" => "FR",
           "postal_code" => "13000",
              "location" => {
            "lat" => 43.2951,
            "lon" => 5.3861
        },
                    "ip" => "2.21.85.4",
         "country_code2" => "FR",
        "continent_code" => "EU",
              "latitude" => 43.2951,
           "region_name" => "Bouches-du-Rhône",
          "country_name" => "France",
             "city_name" => "Marseille",
           "region_code" => "13",
             "longitude" => 5.3861,
              "timezone" => "Europe/Paris"
    },
                     "Dst_IP" => {
         "country_code3" => "IN",
           "postal_code" => "600001",
              "location" => {
            "lat" => 12.8996,
            "lon" => 80.2209
        },
                    "ip" => "23.57.75.218",
         "country_code2" => "IN",
        "continent_code" => "AS",
              "latitude" => 12.8996,
           "region_name" => "Tamil Nadu",
          "country_name" => "India",
             "city_name" => "Chennai",
           "region_code" => "TN",
             "longitude" => 80.2209,
              "timezone" => "Asia/Kolkata"

Output section :

output
        {
                elasticsearch
                                {
                                        hosts => [ "172.27.205.251:9200" ]
                                        index => "logstash_geo"
                                }

                stdout{}
        }

[thomasneirynck]

@Shreesh_Narayanan

The source-and-destination selection in those two dropdowns should both be a different field of type geo_point. It looks like you are using the same field location.

Also a few other things to check:

• make sure that the data is indexed correctly (e.g. use Discover to check you have two geo_point fields with actual lat/lons,
• make sure the time-filter (top right, in Kibana) is large enough for your data-range

[Shreesh_Narayanan]

oh okay, makes sense . To create geo_point for source and destination, i need to have the lon/lat values in a single field,correct ? how do i create it ? i tried adding a field inside geoip such as the one below , but it did not work .should i use a separate mutate filter for this ?

geoip
                {
                        target => "Dst_IP"
                        source => "Forward_IP"
                        tag_on_failure => ["IP-lookup-failed"]
                        add_field => {"Destination_geo" => "[geoip][location][lon]","[geoip]location][lat]"]

                }

[Shreesh_Narayanan]

I managed to get the source and destination lat/lon points in the same field

     "destinationip" => "42.106.161.193",
          "sourceip" => "1.1.1.1",
    "sourcelocation" => "143.2104,-33.494",
      "destlocation" => "88.3832,22.518",

How should the geo_point be created now ? i think it should be for source and destination .

PUT logstashgeotest/_mapping
{
  "properties":{
    "sourcelocation":{
      "type": "geo_point"
    }
  }
}

I tried the above, but elasticsearch (done via dev console) gave an error

{
  "error" : {
    "root_cause" : [
      {
        "type" : "illegal_argument_exception",
        "reason" : "mapper [sourcelocation] cannot be changed from type [text] to [geo_point]"
      }
    ],
    "type" : "illegal_argument_exception",
    "reason" : "mapper [sourcelocation] cannot be changed from type [text] to [geo_point]"
  },
  "status" : 400
}

[Nathan_Reese]

You can not change a field's mapping. You will need to delete the index and then use the correct mappings.

[Shreesh_Narayanan]

thank you . it works now .

sample config where lat and lon values are in the same field and converted to geo point , this conversion was done before any data was pushed/ingested into it

geoip
{
add_tag => [ "geoip" ]
source => "sourceip"
target => "SRC_ip"
#fields => [ "ip", "country_code2", "country_name", "latitude", "longitude" ]
}
geoip 
{
add_tag => ["geoip"]
source => "destinationip"
target => "Dest_IP"
}
mutate 
{
add_field => ["sourcelocation","%{[SRC_ip][longitude]}","tmplat","%{[SRC_ip][latitude]}"]
add_field => ["destlocation","%{[Dest_IP][longitude]}","tmplatdst","%{[Dest_IP][latitude]}"]
}
mutate
{
merge => ["sourcelocation","tmplat"] 
merge => ["destlocation","tmplatdst"]
}
mutate
{
convert => [ "sourcelocation","float"]
convert => ["destlocation","float"]
}
mutate {
remove_field => [ "tmplat","tmplatdst"]
}