Irrelevant field search results

[ethical20]

Hi,

I've made this Lucene query in Kibana to search if Nginx access logs contain a URL with the word "malware":

source.as.organization.name: (/.*[Mm][Aa][Ll][Ww][Aa][Rr][Ee].*/) 

But in Kibana I'm getting the results from other Filebeats fields like

event.dataset:threatintel.abusemalware although I'm expecting to have returned values from source.as.organization.name:

Is this a bug where I search values form a field and get results from other fields? Or is there something wrong from my side?

Regards,

[majagrubic (Maja)]

A few questions:

1. Which version of Kibana are you using?
2. Could you open Inspector in Discover and paste the query part of the request and the highlight part of the response? You should see something like this in the request:

"query_string": {
     "query": "Dest: Reno*",
      "analyze_wildcard": true,
 }

and something like this in the response:

"highlight": {
    "Dest": [
         "@kibana-highlighted-field@Reno Tahoe International Airport@/kibana-highlighted-field@"
    ]
},

[ethical20]

Thanks @majagrubic for getting back.

1. Which version of Kibana are you using?

• Latest 7.16

1. Could you open Inspector in Discover and paste the query part of the request and the highlight part of the response? You should see something like this in the request:

"query_string": {
            "query": "source.as.organization.name: (/.*[Mm][Aa][Ll][Ww][Aa][Rr][Ee].*/) ",
            "analyze_wildcard": true,

"highlight": {
            "fileset.name": [
              "@kibana-highlighted-field@abusemalware@/kibana-highlighted-field@"
            ],
            "event.dataset": [
              "@kibana-highlighted-field@threatintel.abusemalware@/kibana-highlighted-field@"
            ],
            "tags": [
              "@kibana-highlighted-field@threatintel-abusemalware@/kibana-highlighted-field@"
            ]
          }

Your help is appreciated.

Regards,

[majagrubic (Maja)]

Discover is correctly highlight what is receives from an Elasticsearch response. I will move this to the ES forum, perhaps they have more insight into what is happening.

[ethical20]

Ok many thanks, I don't know if I'm right but the highlight should return source.as.organization.name instead of event.dataset right?

Regarding the reply, I should get it here in this ticket right?

Regards,

[majagrubic (Maja)]

That's right, my colleagues should pick it up next.

[ethical20]

Hi @majagrubic,

Any updates on this? Hope we found a solution.

[ethical20]

Hi @majagrubic and all elastic members.

I've updated to the latest 7.16.3 environment but still get same issue. Is this a bug to be reported or is there any help regarding this?

Any help is really appreciated.

[nik9000 (Nik Everett)]

@majagrubic would have to confirm this, but I believe discover uses require_field_match: false on the highlighting config - so that'll highlight all of those terms. Here are the docs for that. In other words - I think kibana is asking for ES to do this. Kibana has a "debug" link in discover that'll show you the search it sent to ES. If it has that parameter in it then it's asking for ES to highlight all of the fields.

If it isn't then maybe it's something to do with the lucene query. I'm not particularly good with those, but I wonder if it should be source.as.organization.name:/.*[Mm][Aa][Ll][Ww][Aa][Rr][Ee].*/ - without the space between the name and the term. I don't the query string query syntax much - I tend to use explicit queries for this sort of thing. Kibana let's you build those as "filters" I think. So, like, if you make a filter on source.as.organization.name of type regexp and pass that regex that'll be more explicit.

[system]

7.1 is EOL and no longer supported. Please upgrade ASAP.

(This is an automated response from your friendly Elastic bot. Please report this post if you have any suggestions or concerns )

[ethical20]

Thanks @nik9000

I think the removing "space" from the query solved part of the problem.

Thank you!