Parsing on a field in Kibana

Elastic Stack Kibana
1

Hi, I am trying to parse out values from a field in Kibana to get a unique count of IDs, but I am unable to parse out this information.

Field Value: I, [2018-02-28T14:50:57.606764 #9] INFO -- : [71f1707b-f78b-4112-a7ae-4437b67b74b1] {"id":"123","name":"abc","db_id":52,"system_id":72}.

How would I be able to parse this information out and get a unique count of IDs per timestamp?

Thanks!
Tony

2

Ideally you should parse the data before inserting it in to Elastic any other option will be very expensive and probably not exactly what you want

However: You can look at scripted fields in kibana but this is going to be slower
https://www.elastic.co/guide/en/kibana/current/scripted-fields.html

if you have a set list of IDs IE you jsut want to know a couple of specific id you can use Filters when doing a table or line graph
https://www.elastic.co/guide/en/kibana/current/xy-chart.html

3

Hi Ed,

Thanks for your explanation. Does the scripted fields only work with JSON objects? For example the value in my "log" key is "I, [2018-02-28T14:50:57.606764 #9] INFO -- : [71f1707b-f78b-4112-a7ae-4437b67b74b1] {"user_id":5, ....}". There is a timestamp before the json object for this key and when I am tried to parse it out using scripted fields I get an "non-array type" error.

"script_fields": {
"test1": {
"script": "params['_source']['log']['user_id']"
}
}

4

Well the document is in Elastic so it is a JSON object. After that "Painless" is a programming language and you can use it to manipulate the data anyway you want. But you should probably open a thread about how to use script_fields as I am not very familiar with the details. Just a couple basic ones and found for my needs it was too slow and pre-parsing was easier and faster

5

We should also mention that unique counts in elastic search (for indexes with multiple nodes/shards) are just an estimate. More information here

Logically it makes sense - cause the only way to determine if a value is truly unique is to have all of the data in one spot for analysis - this is an example of the compromise between speed and accuracy.

This is one thing people moving from relation databases often struggle with (and when users with no understanding of how elastic search is working under the hood see "unique count" in Kibana they assume its a relational DB level of accuracy).

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.