Elastic search machine learning/ pattern recognition version 7.10.2

Elastic Stack Elasticsearch
1

Hi, I have a question to find the root cause of a failed test. In 1st step, we have centralized the data in the Elasticsearch(7.10.2) index which contains the test id and the message(log event).
For the failed tests to find the root cause I want to check if a few lines appear in the logs message for a specific test and if appear it should tell the reason for failure accordingly (ie. a custom failure message).

So for this, I want some directions, How can I achieve this? I am new to this topic. Looking forward to the feedback.
Thank you.

2

It's not apparent here if you really need to use ML to analyze the logs or if you simply accomplish this with a search (i.e. search for the occurrence of the failed test, note the time and test id, then search in the logs for that test id and timeframe of the test failure).

3

Thank you @richcollier for the reply. I want to use ML to analyze the logs. I can elaborate on it. I have 5 tests running and for each test after the execution, it sends logs to elastic search. When a test fails the logs belonging to that test need to be analyzed using ML to give a specific reason on GUI that why it failed.
Each test pushes around 1000 events after success or failure and there are 5 to 6 or sometimes more events give very meaningful information about the failure so reading that I want to give the root cause for the failed test.
so for that, I need directions to achieve it, I am very new to this topic. Thank you.

4

You could investigate using Categorization which allows you intelligent analysis of unstructured logs (if that's truly what you have). Keep in mind the video explains the concepts but the UI has changed slightly since the recording.

5

Thank you @richcollier that looks great, can you please also give some insight if it's the use case of elastic search (I mean is this possible) that reading the logs and giving the custom message root cause for the failed test? I am using 7.10.2 version.
assuming that categorization is done and category 2 has all the FAIL events, so reading that predicts the failure reason?

6

Categorization wouldn't conclude that a particular message type caused the failure, it would show the user that at the time of the failure, which particular message types (categories) behaved differently than their historical behavior (i.e. a spike in the count of a certain message type or the sudden appearance of rarely seen message types that haven't been seen before).

Whether or not this solves what you want to accomplish remains to be seen.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.