Help required in visualization

Elastic Stack Kibana
1

Hello,

I do have a requirement to create visualizations of all total transactions and transactions that contain "Error" or "Exception" anywhere. The whole message is saved as {"type" : "text"} as per the mapping. Each transaction would contain different data structure so I would not be able to change to {"type : Keyword"} since I read it might cause performance issue (correct me if I'm wrong) and the other solution was to turn on field data which is also not advisable. Is there any other solution, please let me know?

P.S - My transactions are XML, but stored as String right now (using store_xml => false). Can this issue be solved somehow if I store transactions as XML?

Cheers,
Maadavan

2

Hello,

Can you show an example of a document, how it looks in ES/Kibana? Type: text is the proper mapping so that shouldn't be a problem. If you search for "Exception" in discover does it show the proper documents?

3

@Marius_Dragomir,

Thanks for swift reply.

Unfortunately would not be able to share Kibana screenshot due to organisation policy. And yes, I am able to get the desired results in discover with below query.

> message : *Error* or message : *Exception*

But in Visualizations, "message" field is not available since it is not a keyword, but a text as per mappings.

Would you be able to help with these limited info. please?

Cheers,
Maadavan

4

You should use the multi-field setup in Elasticsearch in order to have the field both as raw and as keyword. This way you can search for it in Discover and also use it in Kibana visualizations.
https://www.elastic.co/guide/en/elasticsearch/reference/current/multi-fields.html

5

@Marius_Dragomir,

Few queries.

  1. The "message" field which we are discussing here, would be big up to 5MB, so setting this multi field wouldn't cause any performance issue?
  2. Since the field would contain many values, is it safe to increase from default ignore_above to values around 100000.

"ignore_above": 256

  1. Below is the snippet of current mapping, I dont see find exactly where it has to be changed to make it as multi-field based on the reference you shared.

"message": {
"type": "text",
"fields": {
"keyword": {
"type": "keyword",
"ignore_above": 256
}
}
}

Cheers,
Maadavan

6

I have changed my indexing slightly by trimming few unwanted wrappers in the message and adding it to a new field "payloadString" which is now available for aggregation , but not for most of my documents, since it is ignored due to below parameter.

ignore_above : 256

So is there any other solution to make entire string available for aggregation?

Cheers,
Maadavan

7

Team,

Any suggestions or idea please?

Cheers,
Maadavan

8

Kibana Team,

Any suggestions please?

Cheers,
Maadavan

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.