Search for specific data in Kibana UI

Elastic Stack Kibana
1

I got some data here that I would like to simplify in the Visualization section:

cs(Refer): ...orderid=33065759....

I have this that works for just that specific query

cs\(Referer\):"orderid=33065759"

but if I take away a number, lets say the 9 at the end, then Kibana can no longer find the data, but the query should still be true. I thought it might be an issue with the "=" in there, but I tried putting \ before it, but I still have no luck.

I am trying to find specific "orderid=*" sequences like "orderid=3*" "orderid=4*" but if I include or exclude the wildcard, no data is returned though the statement orderid=3 should still be true.

Any help would be greatly appreciated.

Thanks

2

I believe the problem is your use of quotes. When you use quotes, you're telling Lucene to match the exact term. Something like cs\(Referer\):orderid=33* should work.

3

Unfortunately that does not work either.

Here is an example that displayed when using that query

cs\(Referer\):orderid=3*

returned

fuseaction=store.PartInfo&PartNumber=2105030201&VehicleID=35601&diagram=5716090&Title%3D-E320-E430-E55-PROTECT-SHIELD

4

Oh, maybe I made a bad assumption about what the values in the cs(Referer) field look like.

Is fuseaction=store.PartInfo&PartNumber=2105030201&VehicleID=35601&diagram=5716090&Title%3D-E320-E430-E55-PROTECT-SHIELD an example of the full value of that field? And is the field analyzed in Elasticsearch?

5

That is an example of what was returned when I queried

cs\(Referer\):orderid=3*

as you can see in there, there is no mention of "orderid"

In theory, that line of data should not be shown because it does not contain "orderid". Does that make sense?

6

I suspect this is an Elasticsearch thing, which is why I asked if the field is analyzed. Because that field is analyzed (ie. chunked up into searchable terms), the query is too. So you enter orderid=3* as you query, which looks like a word/value to you, but Elasticsearch analyzes that value and turns it into something like orderid and 3 (I don't know if that's the exact values that come out, but hopefully you get the idea). So the query is actually saying "Find me values that match any of these terms", and because that string has a 3 in it, it's a match.

TL;DR - check the field mapping, and if it's analyzed, create a non-analyzed version of the field too, and search on that instead.

7

Did you check in the results what text was highlighted?
That would tell you what Kibana/ES actually found.

As @Joe_Fleming wrote, the = sign probably breaks up orderid and 3* into two different words.
Also, 3 and 3* are different, the first looks for the exact "word" 3 while the latter looks for all words that start with 3.

1 Like
8

@Joe_Fleming the field is analyzed. I cannot make a field not analyzed and use that data for visualizations to my knowledge. It appears that either Kibana or Elasticsearch ignore a lot of special characters like = and appears to treat it like a space

@atira In the Discover tab, the = is not highlighted, but the orderid is and if I type in the whole number afterwards, that will show as well. However, the * will not work much like you were saying since that = is in there. I tried putting \ in front of the equal sign, but that did not help either. If I know the entire number after orderid, I can put it in quotes like cs\(Referer\):"orderid=33062951" but it will not show the expected data with cs\(Referer\):"orderid=3*" or cs\(Referer\):"orderid\=3*" or cs\(Referer\):orderid\=3*

I am not a pro with regular expressions, so if anyone has one that would work in my case, it would be appreciated.

9

Elasticsearch breaks up the string into words and some characters are practically handled as delimiters.

The string orderid=33062951 is broken up into: ["orderid", "33062951"]
You see no equal sign there.

When you query cs\(Referer\):"orderid=33062951" you are pretty much telling Kibana to look for this:

cs\(Referer\):"orderid OR 33062951"

I don't know how the field cs\(Referer\) looks like but if there are no other 8 digit sub-strings in the string, you can just leave the orderid and look only for the number.

Wildcards like the asterisk work within words, not on the whole field.

I know it's confusing, I'm getting used to it myself too. What I've found confusing: / is a delimiter, but _ is not. You can see that by checking highlighted text in results. Delimiters are never highlighted, because they can't ever be part of the results.

This is all handled by ES's analyzer, which is a topic I don't know much about, so I'll have to stop here.
Read here.

10

I get what you are saying. I believe it is reading the delimiter as an "and" in quotes and "or" outside of quotes. Searching for orderid=33062951 will return anything with orderid or 33062951, but if I put it in quotes "orderid=33062951" it is only shows items that contain both orderid and 33062951. The asterisk doesn't work either way, I imagine by the way you described. The question mark in place of a single character doesn't work either i.e. orderid=3???????

11

I cannot make a field not analyzed and use that data for visualizations to my knowledge.

Yes and no. You're right that you probably want the field values to be analyzed for many cases, but you can store both. Check out the docs on multifields, which allow you to store values both ways. Logstash does this, adding a .raw version of string fields to store their non-analyzed values. If you do the same, you can query the raw value with something like cs\(Referer\).raw:"orderid=33062951", which will return only exact matches.

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.