Filtering in Kibana

Elastic Stack Kibana
1

Hi folks,

I'm having a weird filtering result with Kibana, so I want to filter my output using NOT messsage: ABC which shows nothing.

But when I do the opposite way messsage: ABC kibana shows the output of messages containing ABC only.

So when I search based on a=b it works but with a != b it doesn't ...

Is there any other alternative using regexp or other for this case? I just want to see messages NOT containing ABC

Thanks,

2

Hi @OucemaBellagha,

that sounds unusual indeed. message:ABC and NOT message:ABC should result in non-overlapping sets of documents with the number of matches with NOT message:ABC plus the number of matches message:ABC should equal the number of matches with *. Can you confirm that this is not the case?

3

If you are filtering using the search bar, have you tried -message:ABC ?

4

Yeah this is a bit strange so e.g: I have these data for *:
-sdabcd
-sds
-abcde
When I apply the filter message: ABC:
-sdabcd
-abcde
this make sense since abc exists on -sdabcd and -abcde
When I apply the filter NON message: ABC:
Nothing shows up, but normally I should see -sds since it doesn't contain abc

5

Doesn't work.

6

I assume you're using these strings as simplified examples for your actual data. While useful to explain the issue, this makes it hard to diagnose problems with the query that might be caused by the way the data are analyzed during indexing. If you could provide example values, that reflect the shape of the field values more accurately, it might ease the investigation.

7

I want to extract SyscheckFile.path containing /App_Data from my search so :

  • filter shows 422 output

SyscheckFile.path://App_Data filter shows just 9 output

NOT SyscheckFile.path://App_Data filter shows nothing

when it should actually shows 422-9=413 output

8

You're probably colliding with the regexp query syntax here. Try escaping the / with backslashes, as in SyscheckFile.path:\/App_Data. (In addition, for this to work correctly the SyscheckFile.path field should not be analyzed, because the default string analyzer will tokenize the value and discard puctuation like the /.)

9

Already tried /App_Data and *(/)*App_Data * and -SyscheckFile.path://App_Data and so forth but it seems to be strange since the same filter applied without NOT is working but when I add NOT it doesn't work, so this is mainly the problem and it can be an issue related to kibana itself since the search is translated to a Lucence query.

10

The query is sent to the Elasticsearch api as a query string query. No translation is happening there. What is the type of the SyscheckFile.path field in the mapping?

11

"type": "phrase"
But still doesn't explain the NOT filtering, and actually it can be related to that "/" but the thing is that it's working when I just apply SyscheckFile.path://App_Data

12

That is because the unescaped // represent an empty regex, which is matched by any string.

What I meant with the field type is the type defined for the SyscheckFile.path field in the Elasticsearch mapping of the index containing the documents. You can see the mapping by executing curl -H 'application/json' ' ${YOUR_ELASTICSEARCH_API}/${YOUR_INDEXNAME}/_mapping/*/field/SyscheckFile.path.

13

In the mapping "type" : "string"

14

Since the data in that field are filesystem paths and not normal text, I would recommend setting the type of the field to keyword (If you are using Elasticsearch 5.x or later) or set "index": "not_analyzed" for earlier Elasticsearch versions. This would prevent the default analyzer from incorrectly tokenizing the path and should enable prefix queries like the one you're attempting to perform.

See this Elastic blog post for the differences between 5.x an earlier versions.

15

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.