Maps showing incorrect data

[bm11100 (Brent Murphy)]

Hello,

I'm using the o365 integration and want to save a map of source login locations.

When I edit the map, the data is correct, see below

Once the map is saved, the data is incorrect, it has many more data points that do not match the logs, see below

I am using v7.16.2 of ES, Kibana and Agent

[thomasneirynck]

hi @bm11100

Can you take a look at the global time-selection when you opened the saved map (top-right).

It's possible to that the time-selection is larger, catching more data-points.

[bm11100 (Brent Murphy)]

The time on the dashboard and the time when editing the map are the same, but the results are different on the dashboard than when editing. Nothing with the index or query changed.

Dashboard example

Editing map example

[thomasneirynck]

thx @bm11100

vexing!

One difference jumps out to me:
On the Dashboard, there is a big cluster in the middle of Kansas. This is an infamous artifact of geocoding with IP (e.g. How an internet mapping glitch turned a random Kansas farm into a digital hell)

What happens when you remove the map from the dashboard, and then add the map again as a new saved map? Does the discrepancy still show?

Another thing to check if the map queries are indeed the same, on the dashboard vs the map.

Do this by opening the inspector:

On dashboard, open in edit-mode and select the "Inspect" option in the action-menu for the map panel

On Map, click "Inspect" top right.

The Inspector lists all the queries and responses from Elasticsearch. They should be identical.

[bm11100 (Brent Murphy)]

The problem persists when saving as a new map and adding back to the dashboard. The inspect elements seem to be pretty identical.

Dashboard

{
  "size": 0,
  "track_total_hits": false,
  "aggs": {
    "gridSplit": {
      "geotile_grid": {
        "bounds": {
          "top_left": [
            -146.25,
            48.9225
          ],
          "bottom_right": [
            -45,
            21.94305
          ]
        },
        "field": "source.geo.location",
        "precision": 7,
        "size": 65535,
        "shard_size": 65535
      },
      "aggs": {
        "gridCentroid": {
          "geo_centroid": {
            "field": "source.geo.location"
          }
        }
      }
    }
  },
  "fields": [
    {
      "field": "@timestamp",
      "format": "date_time"
    },
    {
      "field": "event.ingested",
      "format": "date_time"
    },
    {
      "field": "file.mtime",
      "format": "date_time"
    }
  ],
  "script_fields": {},
  "stored_fields": [
    "*"
  ],
  "runtime_mappings": {},
  "_source": {
    "excludes": []
  },
  "query": {
    "bool": {
      "must": [],
      "filter": [
        {
          "bool": {
            "must": [
              {
                "exists": {
                  "field": "source.geo.location"
                }
              },
              {
                "geo_bounding_box": {
                  "source.geo.location": {
                    "top_left": [
                      -146.25,
                      50.73646
                    ],
                    "bottom_right": [
                      -42.1875,
                      21.94305
                    ]
                  }
                }
              }
            ]
          }
        },
        {
          "range": {
            "@timestamp": {
              "format": "strict_date_optional_time",
              "gte": "2022-01-05T07:00:00.000Z",
              "lte": "2022-01-06T06:59:59.999Z"
            }
          }
        }
      ],
      "should": [],
      "must_not": []
    }
  }
}

Map

{
  "size": 0,
  "track_total_hits": false,
  "aggs": {
    "gridSplit": {
      "geotile_grid": {
        "bounds": {
          "top_left": [
            -157.5,
            55.77657
          ],
          "bottom_right": [
            -33.75,
            11.1784
          ]
        },
        "field": "source.geo.location",
        "precision": 7,
        "size": 65535,
        "shard_size": 65535
      },
      "aggs": {
        "gridCentroid": {
          "geo_centroid": {
            "field": "source.geo.location"
          }
        }
      }
    }
  },
  "fields": [
    {
      "field": "@timestamp",
      "format": "date_time"
    },
    {
      "field": "event.ingested",
      "format": "date_time"
    },
    {
      "field": "file.mtime",
      "format": "date_time"
    }
  ],
  "script_fields": {},
  "stored_fields": [
    "*"
  ],
  "runtime_mappings": {},
  "_source": {
    "excludes": []
  },
  "query": {
    "bool": {
      "must": [],
      "filter": [
        {
          "bool": {
            "filter": [
              {
                "bool": {
                  "should": [
                    {
                      "match_phrase": {
                        "event.action": "UserLoggedIn"
                      }
                    }
                  ],
                  "minimum_should_match": 1
                }
              },
              {
                "bool": {
                  "must_not": {
                    "bool": {
                      "should": [
                        {
                          "query_string": {
                            "fields": [
                              "user.id"
                            ],
                            "query": "Sync_*"
                          }
                        }
                      ],
                      "minimum_should_match": 1
                    }
                  }
                }
              }
            ]
          }
        },
        {
          "match_phrase": {
            "event.action": "UserLoggedIn"
          }
        },
        {
          "bool": {
            "must": [
              {
                "exists": {
                  "field": "source.geo.location"
                }
              },
              {
                "geo_bounding_box": {
                  "source.geo.location": {
                    "top_left": [
                      -157.5,
                      55.77657
                    ],
                    "bottom_right": [
                      -30.9375,
                      8.40717
                    ]
                  }
                }
              }
            ]
          }
        },
        {
          "range": {
            "@timestamp": {
              "format": "strict_date_optional_time",
              "gte": "2022-01-05T07:00:00.000Z",
              "lte": "2022-01-06T06:59:59.999Z"
            }
          }
        }
      ],
      "should": [],
      "must_not": []
    }
  }
}

[jsanz (Jorge Sanz)]

The map request has a whole filter not appearing in the dashboard query. Is that query coming from the Kibana search bar or it is set up in the layer definition?

[bm11100 (Brent Murphy)]

Maps on the dashboard don't save the filters applied when creating it?

[Nathan_Reese]

Maps on the dashboard don't save the filters applied when creating it?

That is correct. There is an open issue for this [Maps] Filters applied to map visualization not preserved when added to dashboard · Issue #91282 · elastic/kibana · GitHub. Please add any comments about your use case.

[Nathan_Reese]

For a work around, you can apply your filter to the individual layer in your map. That way the filter will also be applied in Dashboard

[bm11100 (Brent Murphy)]

Thanks for the workaround! I didn't think to set it in the layer itself.