Map data not displaying in 8.2.0

Hi,
Ive migrated from 7.16 to 8.2 but since then my maps are not displaying.
The data is there and mapped with a gep_point for the location. I also noticed the location is no longer displaying in the discover ie with the world symbol next to it. I guessing this is part of the problem. If I try to create a new map it detects the geo location field despite it not showing in the discover. It shows correctly in the data view/index pattern.
Ive looked through the release notes but cant see what might have broken it.
Any help would be appreciated. The data is ingested via logstash.

Thanks

Are you running in cloud or on-prem?
What version are you running? You said 8.2 but is that 8.2.0 or 8.2.1, etc.
When adding a "Documents" layer, does the data display when you change scaling to "limit results to 10000"?

Hi Nathan,

Thanks for the reply. Using 8.2.0,
No the data doesn't show when you limit to 10,000. Think that's default anyway.
On Prem version

Would you expect the world symbol to show in the discover for geo data or has that changed in 8.2? Thats the only thing i can see is different from 7.16, mapped as geo_point, maps doesn't complain ie saying no geo data found etc.
No log errors. Put the same data back in to 7.16 and shows correctly. All other data in the environment shows correctly too, just the geo data not displaying.

Thanks

In console can you run the following command GET /your_index_name/_mapping, replacing 'your_index_name'. What are the results. What is geo field mapped as?

Hi Nathan,

Here what it returned, see below. it looks a bit messy as that's not what's in the template file i.e some additional fields like country_iso_code , is there some auto mapping occurring?:

partial extract:

"threatindicator" : {
          "properties" : {
            "city_name" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "continent_code" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "country_code2" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "country_code3" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "country_name" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "dma_code" : {
              "type" : "short"
            },
            "geo" : {
              "properties" : {
                "city_name" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "continent_code" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "country_iso_code" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "country_name" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "location" : {
                  "properties" : {
                    "lat" : {
                      "type" : "float"
                    },
                    "lon" : {
                      "type" : "float"
                    }
                  }
                },
                "postal_code" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "region_iso_code" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "region_name" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "timezone" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                }
              }
            },
            "ip" : {
              "type" : "ip"
            },
            "latitude" : {
              "type" : "half_float"
            },
            "location" : {
              "type" : "geo_point"
            },
            "longitude" : {
              "type" : "half_float"
            },
            "mmdb" : {
              "properties" : {
                "dma_code" : {
                  "type" : "long"
                }
              }
            },
            "postal_code" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "region_code" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "region_name" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "timezone" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            }
          }
        },

thanks

Martin

@martb

perhaps take a look at this...

@Nathan_Reese is this something different?

There are similarities but before i investigate further did the fix not get pushed into 8.2 or have i misread?

I appreciate you getting back to me on this. Thank you.

@martb
Did you simply try to set the scaling not to vector tiles and see if that worked as shown as a workaround ? That's a 10 second check?

And would help confirm whether the fix is in or not?

I'll validate with a self-managed when I get home later today

That fix appears to be in 8.2.2, this would not have worked previously... I will do a quick 8.2.0 check as well.

Confirmed that fix is in 8.2.0, so that appears to not be the source of your issues.

Apologies for the side track

I do notice that there are 2 different location fields one is a geo_point the other is not... is that intentional?

Hi,

I tried that but not difference.

The differences from my previous working version seems tobe the new geo fields that are now showing in the mapping template.
Ive seen the same fields in the ECS. I did a test to change my mappings to match the new fields but this didnt make any difference.

The other thing that is jumping out at me is no geo field showing in the Discover ie the world symbol, it's detected in maps, but not showing in Discover.
I've been using elastic for over 4 years and I know previously maps with logstash required you to define and set alot more in logstash and mapping. Is 8.x automatically setting the geo fields and therefore what's defined in my existing mapping is conflicting with the auto geo fields? Just a theory. Thanks Martin

Our messages crossed. The 2 locations fields are not intentional, also duplicate country fields etc. These seem tobe auto adding.. im now wondering if i simply delete my mapping template and see what is set by default. I suspect the duplicates will disappear.

We are starting to converge a bit.

I was going to suggest that... and if you do you should run filebeat setup -e again to load all the proper templates etc.

Did you upgrade filebeat to 8.2.0 as well?

Which module is this?

HI,

I'm quite sure filebeats would have been upgraded too, but I'll have to confirm that tomorrow too be 100%.

After ingesting the data without any predefined mappings the following was created. As expected there are no duplicates and all look a bit cleaner albeit IP not defined correctly(easy fix).
As no geo_point was set Maps nolonger sees a geospatial field. Location is now added with lat and lon. Previously I set location as: "location" : { "type" : "geo_point" }. I'm now not sure what should be set as a geo_point.

        "threatindicator" : {
          "properties" : {
            "geo" : {
              "properties" : {
                "city_name" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "continent_code" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "country_iso_code" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "country_name" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "location" : {
                  "properties" : {
                    "lat" : {
                      "type" : "float"
                    },
                    "lon" : {
                      "type" : "float"
                    }
                  }
                },
                "postal_code" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "region_iso_code" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "region_name" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "timezone" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                }
              }
            },
            "ip" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "mmdb" : {
              "properties" : {
                "dma_code" : {
                  "type" : "long"
                }
              }
            }
          }
        },

Many thanks

Martin

Just removing the template won't work for sure... You will need to run setup.

Because then you are just getting the default mapping.. which won't be correct.

What makes this even more difficult is if you remove the mapping / template and there's already filebeats out there writing. They will immediately write a document and you'll get the default mapping again and be right back is the same place.

You most likely need to run setup again get the right template and then ingest pipeline loaded.

Order I would do it...

  • Stop all the beats
  • Clean up bad indices / data streams
  • Run setup
  • Start the beats again

Also
7.x beats write indices
8.x beats write data streams

Hi,

Just to clarify, I did understand that it wouldn't work when I removed the template, this was just to see why we had duplicate mappings. I doubled checked regarding Filebeat and it's not installed. I wasn't sure regarding the filebeat reference as we don't use it. Out data is coming in via Logstash connecting a MSSQL database.

Regards

Martin

HI,

I found what the problem is. I'm not 100% sure when the change was introduced, possibly 7.17 but if you don't set ecs_compatibility => disabled in your logstash file(filter) you will have conflicting mappings. Basically you end up with mixture of ECS geo mappings and your existing defined mappings.

To resolve this:

filter {
  geoip {
    source => "[host][ip]"
    ecs_compatibility => disabled
  }
}

Ref:
https://www.elastic.co/guide/en/logstash/7.17/ecs-ls.html
https://www.elastic.co/guide/en/logstash/current/plugins-filters-geoip.html

thanks for you help

Martin

1 Like

Arg! ecs_compatability good find!, that has bit a few other folks too in other areas... should have thought of that.