Kinaba pattern or using a machine learning?

Elastic Stack Kibana
1

Hello Everyone,

I am evaluating the Machine Learning capabilities in Kibana for our error log analysis and visualization. If the system can handle our requirements, we plan to purchase a subscription for the company. I'm reaching out for assistance on a few specific points related to my anomaly detection job setup.

Current Situation: Our error logs have varied structures, which sometimes include unique identifiers like process IDs (e.g., ProcessID). This variability causes individual errors with different process IDs to be counted as separate entries, instead of being grouped by pattern. We aim to use machine learning to identify consistent patterns across these logs, disregarding unique identifiers like process IDs, to visualize the most frequent error types accurately.

Actions Taken:

  • I created an anomaly detection job using the Categorization feature on the message field.
  • I specified a categorization field in the job settings, validated the job, and enabled it in real-time.
  • Despite reproducing errors with unique identifiers (like varying ProcessID values), I am not seeing grouped anomalies for these errors in the results.

I already finish this to step 5

image
image1575×928 39 KB

Challenges:

  1. The job does not appear to be detecting anomalies based on the pattern of logs, even when reproduced in real-time.
  2. The categorization is not grouping logs with similar patterns but different process IDs as expected.

Objective: My goal is to identify the main error patterns across logs and visualize them, with variations such as process IDs ignored by the ML job, so they are counted as part of the same error pattern.

Request for Assistance: Could you guide me on:

  • Fine-tuning the anomaly detection job to categorize error patterns effectively, regardless of unique identifiers.
  • Configuring categorization or data preprocessing to disregard fields like ProcessID in pattern grouping.

My sample visualization

image
image1878×636 64.2 KB

Any insights or best practices for achieving accurate pattern detection with our log structure would be greatly appreciated!

Thank you for your support.