Create and show aggregated data

Elastic Stack Kibana
1

Hi Team,
I have ELK setup in my VMs and I am able to see application logs in kibana. Now I want to aggregate data based on correlation Ids and then when user clicks on any such correlation ID it should show details of the logs for that correlation id(Basically like a hyperlinks where the correaltion ids will be showed and on click of the link details of those corrids should be seen). Can someone please help me if this is possible in Kibana and if yes how can I achieve it?

Any help is greatly appreciated!!

Thanks and Regards,
Anuj Jain

2

I would suggest to put your visualization, with aggregated data on your dashboard and next to it put a saved search which will be showing all your records formatted in a way you like.

clicking on id in visualization will then filter the saved search to only that id.

you can save your search in the discovery app and then use it on your dashboard just as you would visualization.

3

Hi Peter,
Thank you so much for responding. I am very new to Kibana. Can you please give me some example how can I do it? I have app logs in message attribute which contains all the info like correlation id and other details. How Can I aggregate all the logs from the message attribute in elastic and display in Kibana.

Really appreciate your reply .

Thanks and Regards,
Anuj Jain

4

I dont fully understand your question.
Do you mean you have just one field with all the information inside ? If that is the case you should first index your data in a way that correlation id is a separate field, else you will not be able to aggregate on it at all.

Take a look at logstash documentation on how you can parse your logs: https://www.elastic.co/guide/en/logstash/current/index.html

and if you have issues with that try asking in the logstash forum:

if your data is already indexed correctly, please try to explain better what exactly would you like to achieve

5

Hi Peter,
Thank you so much for the response. Please find a json attribute as below: -

"message": "[2017-11-23 02:13:17.513] - debug: [corrId: xxxxxxxx][Response] {\"id\":\"EC-xxxxxx\",\"state\":\"APPROVED\",\"cart_id\":\"EC-XXXXXXXXXXX\",\"shipping_address\":{\"recipient_name\":\"XXXXXX\",\"id\":\"XXXXXXXXX\",\"line1\":\"XXXXXXx, XXXX\",\"line2\":\"XXXXX\",\"city\":\"xxxxx\",\"state\":\"xxx\",\"postal_code\":\"xxxxx-100\",\"country_code\":\"xx\",\"normalization_status\":\"UNKNOWN\",\"type\":\"GIFT\",\"default_address\":true,\"preferred_address\":false,\"primary_address\":false,\"disable_for_transaction\":false},\"payer\":{\"payment_method\":\"paypal\",\"status\":\"UNVERIFIED\",\"payer_info\":{\"email\":\"buyer@plus.paypal.com\",\"first_name\":\"xxxx\",\"last_name\":\"xxxx\",\"payer_id\":\"xxxxxxx\",\"shipping_address\":{\"recipient_name\":\"xxxxx\",\"id\":\"xxxxxx\",\"line1\":\"xx. xx, xxx\",\"line2\":\"apt xxx\",\"city\":\"xxx\",\"state\":\"xx\",\"postal_code\":\"xxxx-xx\",\"country_code\":\"xx\",\"normalization_status\":\"UNKNOWN\",\"type\":\"GIFT\",\"default_address\":true,\"preferred_address\":false,\"primary_address\":false,\"disable_for_transaction\":false},\"phone\":\"+xx xxxxxx\",\"phone_type\":\"HOME\",\"tax_id_type\":\"xxxxx\",\"tax_id\":\"xxxxxx\",\"country_code\":\"xx\",\"user_type\":\"GUEST\"},\"funding_option\":{\"id\":\"xxxxx\",\"rank\":0,\"funding_sources\":[{\"funding_mode\":\"INSTANT_TRANSFER\",\"funding_instrument_type\":\"PAYMENT_CARD\",\"amount\":{\"value\":\"xx\",\"currency\":\"xx\"},\"soft_descriptor\":\"AUTOMETALRE\",\"payment_card\":{\"id\":\"xxxxxx\",\"type\":\"xxxx\",\"number\":\"xxx\",\"card_product_class\":\"UNKNOWN\"}}]},\"funding_options\":[{\"id\":\"xxxx\",\"rank\":0,\"funding_sources\":[{\"funding_mode\":\"INSTANT_TRANSFER\",\"funding_instrument_type\":\"xxxx\",\"amount\":{\"value\":\"xxx\",\"currency\":\"xxx\"},\"soft_descriptor\":\"xxx\",\"payment_card\":{\"id\":\"xxxx\",\"type\":\"AMEX\",\"number\":\"xxx\",\"card_product_class\":\"UNKNOWN\"}}]}]},\"payment_approved\":xxx}"

From the above such attribute I need to get the corrID value and aggregate data based on that corrId and create a hyper link where if any one clicks it will display data related to that corrId. I know we can do a string search that will display the data related to corrid. But what I need is I need to group data based on corrid and display all the available corrids in a particular time frame as hyper links and when any such hyperlink is clicked it will display messages related to that corrid.

Please let me know if you need more information.

Thanks and Regards,
Anuj Jain

6

That is a string containing a JSON document. In order to analyse based on those fields they need to be indexed separately, so you need to parse this field before indexing the data. There are a number of options available:

7

Hi Christian,

Thank you so much for the response.
I will try to explore the options you mentioned. After Indexing how can I aggregate and view data in Kibana?

Really appreciate your help on that.

Thanks and Regards,
Anuj Jain

8

Have a look at the links provided in the getting started guide.

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.