Map data not displaying in 8.2.0

martb · June 16, 2022, 8:27pm

Hi,
Ive migrated from 7.16 to 8.2 but since then my maps are not displaying.
The data is there and mapped with a gep_point for the location. I also noticed the location is no longer displaying in the discover ie with the world symbol next to it. I guessing this is part of the problem. If I try to create a new map it detects the geo location field despite it not showing in the discover. It shows correctly in the data view/index pattern.
Ive looked through the release notes but cant see what might have broken it.
Any help would be appreciated. The data is ingested via logstash.

Thanks

Nathan_Reese · June 16, 2022, 9:21pm

Are you running in cloud or on-prem?
What version are you running? You said 8.2 but is that 8.2.0 or 8.2.1, etc.
When adding a "Documents" layer, does the data display when you change scaling to "limit results to 10000"?

martb · June 17, 2022, 8:54am

Hi Nathan,

Thanks for the reply. Using 8.2.0,
No the data doesn't show when you limit to 10,000. Think that's default anyway.
On Prem version

Would you expect the world symbol to show in the discover for geo data or has that changed in 8.2? Thats the only thing i can see is different from 7.16, mapped as geo_point, maps doesn't complain ie saying no geo data found etc.
No log errors. Put the same data back in to 7.16 and shows correctly. All other data in the environment shows correctly too, just the geo data not displaying.

Thanks

Nathan_Reese · June 18, 2022, 5:30pm

In console can you run the following command GET /your_index_name/_mapping, replacing 'your_index_name'. What are the results. What is geo field mapped as?

martb · June 18, 2022, 9:04pm

Hi Nathan,

Here what it returned, see below. it looks a bit messy as that's not what's in the template file i.e some additional fields like country_iso_code , is there some auto mapping occurring?:

partial extract:

"threatindicator" : {
          "properties" : {
            "city_name" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "continent_code" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "country_code2" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "country_code3" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "country_name" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "dma_code" : {
              "type" : "short"
            },
            "geo" : {
              "properties" : {
                "city_name" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "continent_code" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "country_iso_code" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "country_name" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "location" : {
                  "properties" : {
                    "lat" : {
                      "type" : "float"
                    },
                    "lon" : {
                      "type" : "float"
                    }
                  }
                },
                "postal_code" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "region_iso_code" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "region_name" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "timezone" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                }
              }
            },
            "ip" : {
              "type" : "ip"
            },
            "latitude" : {
              "type" : "half_float"
            },
            "location" : {
              "type" : "geo_point"
            },
            "longitude" : {
              "type" : "half_float"
            },
            "mmdb" : {
              "properties" : {
                "dma_code" : {
                  "type" : "long"
                }
              }
            },
            "postal_code" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "region_code" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "region_name" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "timezone" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            }
          }
        },

thanks

Martin

stephenb · June 19, 2022, 1:26am

@martb

perhaps take a look at this...

@Nathan_Reese is this something different?

martb · June 19, 2022, 6:11pm

There are similarities but before i investigate further did the fix not get pushed into 8.2 or have i misread?

I appreciate you getting back to me on this. Thank you.

stephenb · June 19, 2022, 7:19pm

@martb
Did you simply try to set the scaling not to vector tiles and see if that worked as shown as a workaround ? That's a 10 second check?

And would help confirm whether the fix is in or not?

I'll validate with a self-managed when I get home later today

stephenb · June 19, 2022, 9:07pm

That fix appears to be in 8.2.2, this would not have worked previously... I will do a quick 8.2.0 check as well.

Confirmed that fix is in 8.2.0, so that appears to not be the source of your issues.

Apologies for the side track

stephenb · June 19, 2022, 9:16pm

I do notice that there are 2 different location fields one is a geo_point the other is not... is that intentional?

martb:

                "location" : {
                  "properties" : {
                    "lat" : {
                      "type" : "float"
                    },
                    "lon" : {
                      "type" : "float"
                    }
                  }
                },

martb · June 19, 2022, 9:19pm

Hi,

I tried that but not difference.

The differences from my previous working version seems tobe the new geo fields that are now showing in the mapping template.
Ive seen the same fields in the ECS. I did a test to change my mappings to match the new fields but this didnt make any difference.

The other thing that is jumping out at me is no geo field showing in the Discover ie the world symbol, it's detected in maps, but not showing in Discover.
I've been using elastic for over 4 years and I know previously maps with logstash required you to define and set alot more in logstash and mapping. Is 8.x automatically setting the geo fields and therefore what's defined in my existing mapping is conflicting with the auto geo fields? Just a theory. Thanks Martin

martb · June 19, 2022, 9:24pm

Our messages crossed. The 2 locations fields are not intentional, also duplicate country fields etc. These seem tobe auto adding.. im now wondering if i simply delete my mapping template and see what is set by default. I suspect the duplicates will disappear.

stephenb · June 19, 2022, 9:37pm

We are starting to converge a bit.

I was going to suggest that... and if you do you should run filebeat setup -e again to load all the proper templates etc.

Did you upgrade filebeat to 8.2.0 as well?

Which module is this?

martb · June 19, 2022, 10:11pm

HI,

I'm quite sure filebeats would have been upgraded too, but I'll have to confirm that tomorrow too be 100%.

After ingesting the data without any predefined mappings the following was created. As expected there are no duplicates and all look a bit cleaner albeit IP not defined correctly(easy fix).
As no geo_point was set Maps nolonger sees a geospatial field. Location is now added with lat and lon. Previously I set location as: "location" : { "type" : "geo_point" }. I'm now not sure what should be set as a geo_point.

        "threatindicator" : {
          "properties" : {
            "geo" : {
              "properties" : {
                "city_name" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "continent_code" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "country_iso_code" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "country_name" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "location" : {
                  "properties" : {
                    "lat" : {
                      "type" : "float"
                    },
                    "lon" : {
                      "type" : "float"
                    }
                  }
                },
                "postal_code" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "region_iso_code" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "region_name" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                },
                "timezone" : {
                  "type" : "text",
                  "fields" : {
                    "keyword" : {
                      "type" : "keyword",
                      "ignore_above" : 256
                    }
                  }
                }
              }
            },
            "ip" : {
              "type" : "text",
              "fields" : {
                "keyword" : {
                  "type" : "keyword",
                  "ignore_above" : 256
                }
              }
            },
            "mmdb" : {
              "properties" : {
                "dma_code" : {
                  "type" : "long"
                }
              }
            }
          }
        },

Many thanks

Martin

stephenb · June 19, 2022, 10:57pm

Just removing the template won't work for sure... You will need to run setup.

Because then you are just getting the default mapping.. which won't be correct.

What makes this even more difficult is if you remove the mapping / template and there's already filebeats out there writing. They will immediately write a document and you'll get the default mapping again and be right back is the same place.

You most likely need to run setup again get the right template and then ingest pipeline loaded.

Order I would do it...

Stop all the beats
Clean up bad indices / data streams
Run setup
Start the beats again

Also
7.x beats write indices
8.x beats write data streams

martb · June 20, 2022, 9:23am

Hi,

Just to clarify, I did understand that it wouldn't work when I removed the template, this was just to see why we had duplicate mappings. I doubled checked regarding Filebeat and it's not installed. I wasn't sure regarding the filebeat reference as we don't use it. Out data is coming in via Logstash connecting a MSSQL database.

Regards

Martin

martb · June 20, 2022, 11:55am

HI,

I found what the problem is. I'm not 100% sure when the change was introduced, possibly 7.17 but if you don't set ecs_compatibility => disabled in your logstash file(filter) you will have conflicting mappings. Basically you end up with mixture of ECS geo mappings and your existing defined mappings.

To resolve this:

filter {
  geoip {
    source => "[host][ip]"
    ecs_compatibility => disabled
  }
}

Ref:
https://www.elastic.co/guide/en/logstash/7.17/ecs-ls.html
https://www.elastic.co/guide/en/logstash/current/plugins-filters-geoip.html

thanks for you help

Martin

stephenb · June 20, 2022, 2:12pm

Arg! ecs_compatability good find!, that has bit a few other folks too in other areas... should have thought of that.

system · July 18, 2022, 2:13pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

Topic		Replies	Views
Kibana map - No results Found documents Kibana maps	7	1183	June 1, 2021
Kibana Tile Map not showing geo points Kibana	4	1739	September 20, 2017
Kibana Maps features not displaying (failed net::ERR_CONTENT_DECODING_FAILED) Kibana maps	8	1155	May 13, 2022
Kibana not plotting my Geo_point Kibana	3	669	March 27, 2019
Layer dot found result Kibana maps , runtime-fields	29	1327	September 11, 2022

Map data not displaying in 8.2.0

Related topics