Maps with multiple layer appear not to work as expected

[Russell_Fulton]

Kibaba 7.10

When I create a map with multiple layers and then hide all bar one layer the map always displays the same data when in a dashboard panel:

The major difference between the two layers is that the "all auth" has a large circle for China which disappears when one filters out SSH

In this first image which excludes the SSH auth we still see the big circle.

The other layer is essentially the same image.

When I edit the layers individually they display as expected:

this shows the excl ssh as it should be a small circle for China.

is the all auth data (incl ssh) with the big circle.

What am I missing.

Side note: I have considerable experience with map applications the map layer interface is the worst I have seen on any map application! SE should really have checkboxes anything else is really clunky to use.

[markov00 (Marco Vettorello)]

Hi @Russell_Fulton
have you configured a filter on the Clustered excel ssh to exclude event_source: SSH?
Because if those two layers shows the same count, but when adding a global filter the count is reduced by the SSH events, then you probably have missed adding that filter on the excl ssh layer.

I have considerable experience with map applications the map layer interface is the worst I have seen on any map application! SE should really have checkboxes anything else is really clunky to use.

Thank you for the feedback, it will be great and can help us on improve the interface if you can elaborate a bit more on that

[jsanz (Jorge Sanz)]

Justo to complete @markov00 answer, and sorry if this is not necessary, but Map layers have their own filtering and can be configured to ignore Kibana global search bar.

This is for Kibana 7.10 and the sample flight map.

@Russell_Fulton

the map layer interface is the worst I have seen on any map application!

As Marco also mentioned, we really appreciate your feedback on what you miss from the application and why is not reaching your expectations.

[Russell_Fulton]

@markov00

Thank you for the feedback, it will be great and can help us on improve the interface if you can elaborate a bit more on that

The most common method of method of manipulating layers is with a simple check box to control visibility. This means that you can change the state with a single click of the mouse rather than a click to reveal the popup move the mouse to select the option and then another click. When there are several layers all the extra clicks and mouse movement equals frustration : )

The issue is where one has more than one option for the layer (ES has fit to data) so I can see why it is done this way

[Russell_Fulton]

@markov00

Hi @Russell_Fulton
have you configured a filter on the Clustered excel ssh to exclude event_source: SSH?
Because if those two layers shows the same count, but when adding a global filter the count is reduced by the SSH events, then you probably have missed adding that filter on the excl ssh layer.

yes, That is why I included the individual screen shots from the composer which clearly shows the filter and the filtered result.

so far as I can tell when I show any layer on its own (well with the base layer too!) I don't get the the layer I want I get the layer that at the top of the list.

[Russell_Fulton]

Aside: you can see very clearly where most of the SSH scanning is coming from

[Russell_Fulton]

I have just figured out what the issue is and that is that the filters apply to the map as a whole. You can't have different filters for each layer.

This is not at all clear from the user interface.

What I was trying to do was effectively having 4 maps one in each layer to save space on a dashboard.

I am not sure how one can make this clear in the compose UI.

[Nathan_Reese]

I have just figured out what the issue is and that is that the filters apply to the map as a whole. You can't have different filters for each layer.

You can add filters to individual layers. Filter a single layer | Kibana Guide [8.11] | Elastic

[Russell_Fulton]

Thanks Reese! Back to the drawing board!

In the mean time I have rework this into 4 maps (as opposed to 4 layers one map) have these in 4 separate panels in my dashboard and I still have the same problem that I started with.

All the maps get the same filter one they are put into a panels on a dashboard. I was really starting to doubt my sanity so I got one of my colleagues to sanity check what I have done and he agrees that there is something screwy going on.

Should I be logging an issue on github?

[Nathan_Reese]

Before opening an issue, lets get a better understanding of the problem.

Can you please describe your data set along with mappings. Can you also describe your use case. What problem are you trying to solve? What information are you trying to get from the map. Its hard to tell from the provided screen shots.

It sounds like filters are not getting applied as expected? Please provide the elasticsearch queries from each layer, Troubleshoot Maps | Kibana Guide [7.12] | Elastic. To open inspector for a panel in dashboard, click the icon in the upper right.

[Russell_Fulton]

Ok we will drill down here! : )

forget the complication of layers. I have now established that I have the same problem when I put the maps in separate panels in a dashboard.

simply stated the issue I am having is that when I have more than one map in a dashboard the filters in the maps get messed up.

What I had assumed would happen is that (effectively) I would get the global filter for the dashboard anded with the filter specified in the map.

What appears to be happening is that I get the same filter applied to all maps in the dashboard. This filter is composed as above but just once for one of the panels and so I end up with the same filter for each map.

When I do an inspect on a map (either in the panel or in maps) I don't see the filters at all, which presumably means they are stored separately.

{
  "docvalue_fields": [
    "src_location"
  ],
  "size": 10000,
  "_source": false,
  "stored_fields": [
    "src_location"
  ],
  "script_fields": {},
  "query": {
    "bool": {
      "must": [],
      "filter": [
        {
          "match_all": {}
        },
        {
          "exists": {
            "field": "status"
          }
        },
        {
          "match_phrase": {
            "status": "success"
          }
        },
        {
          "match_phrase": {
            "event_source": "SSH"
          }
        },
        {
          "geo_bounding_box": {
            "src_location": {
              "top_left": [
                -180,
                89
              ],
              "bottom_right": [
                180,
                -89
              ]
            }
          }
        },
        {
          "range": {
            "@timestamp": {
              "gte": "2021-04-03T23:46:21.085Z",
              "lte": "2021-04-04T23:46:21.085Z",
              "format": "strict_date_optional_time"
            }
          }
        }
      ],
      "should": [],
      "must_not": [
        {
          "range": {
            "src_ip": {
              "gte": "130.216.0.0",
              "lt": "130.216.255.255"
            }
          }
        }
      ]
    }
  }
}

What I am struggling with is that if this is a bug the why has no one else has tripped over this long ago But I can't see what I could be doing wrong.

One of my colleagues was able to reproduce the issue, albeit on the same kibana instance.

We need to see the actual request for the maps in the panel complete with the filters!

[Nathan_Reese]

Are you running into Filters applied to map visualization not preserved when added to dashboard?

[Russell_Fulton]

Yes, Until I looked really closely I had not realised that the filters were being lost completely.

I see you have classified it at a "feature" -- a term that I have not heard for a long time to refer to a bug that that isn't a bug. ; )

The behaviour certainly is counter intuitive! The enhancement will make maps much more useful useful in dashboards. In the meantime I will do without them : )

[Russell_Fulton]

LOL! within seconds of posting my last message an email from my colleague arrived in my inbox. He pointed out how it is supposed to be done.

You can set filters on the layers using lucene which also means (in my case) that one can use cidr block notation or IP address ranges! These filters work in the panels.

It also means that my original approach of having one map with 4 layers works too.

[Russell_Fulton]

One last comment now that I (think) I understand what is going on. The only enhancement need in maps is to make it clear in the map composer that there are two types of filters. "layer filters" which are carried with the maps wherever they go and "global" filters which are not logically part of the map.

Simply putting "Global" on the "add filter" button!

and better still replacing the clunky:

panel with one the same as the layer filter:

that we all know and love from Discover! adding some text explaining the significance of "global" and pointing users to the layer filters that I missed completely.

All the necessary functionality is there what is need it some tidying up of the UI so it it more intuitive.