I want to create something like a Machine Learning or Threshold rule that triggers when my infrastructure loses events from elastic agents.
For example, the medium of events per day is 500k logs. But if I will get only 100k for the day, I would like to get a notification/alert that informs me about losing events so I may immediately take a look at what is going on.
I tried to create a new job in Anomaly Detection, but it looks like there is no option to create a rule that works in real-time, not based on a dedicated time range.
Machine Learning jobs are meant for realtime while setting up the job it will ask for a historical time frame to learn from (you can say none / very little).. but a the end, you can tell it to run ongoing and it will alert you on as anomalies going forward.
BTW the log rate analysis job does exactly what you describe.
This is a bit of a wizard...
Hi Stephen! I tried to start with Elastic Observability Anomalies, but I got the error and there are no details I should look for.
When I open the Log rate tab, Elastic loads indexes for a few minutes. Then I see this error and can't create an ML job. The button is unclickable because of this issue.
I wonder what is that kibana_sample_data_logs* index. I may get an error because of this index. Is it possible that the index exists in some config file, but in fact it doesn't exist in Kibana? And when Kibana is trying to load that index - Kibana can't find it and returns an error.
The kibana is sample data that You or someone installed. it can be removed. There is even a button to remove it? If you go to the home screen then add data and then I think it's under sample data then there should be an option to remove it. Or just go to Data views and remove the data view
Is this elastic cloud or a self-managed cluster? And do you have a machine learning node?
Rather than start a whole new topic, I figure I will ask here since it is mildly related: I have gotten the ML categorization working with our (many) log files, but have not found a way to identify what the actual category fields are outside of "mlcategory#". Would this be due to still using 7.17 or am I just missing something very obvious? I felt like there were more discovery options on the trial with 8. Sorry for the basic question, I am pretty new to this and trying to catch up as fast as I can.
Hi, thanks for the response. I am trying to determine the fields associated with the different categories, but don't know where to start... I can look at examples, but have no hint what fields it is alerting on. Maybe to be more clear: I don't know how they are being categorized.
Thanks, I'm really just trying to get a feel for it with our data, understand how it is categorizing, and figure out how it can help me. If I can only understand the different categories (influencers) as "mlcategory_" without knowing how they are being categorized, it leaves me a bit lost. Thanks for the help, I'll keep on reading.
Ok so even simple statement like this you will need to clarify...
Apologies but I am not trying to be difficult but there are so many topics / details with ML and / or Categorization we won't be able to help unless you start at the beginning... You said you are just getting started, I am just trying to help you get and answer...
What data, What Categorization there are many ways to set up categorization ... are you talking about logs? one of the OOTB Jobs? if it is logs with message fields the categorization is based on pattern matching...
This was in an attempt to find out how the different categorization is being determined, so I can find out what is actually going on behind anomalous behaviors associated with the many different "mlcategory_"
Seriously, thank you for the help . I am also fine dropping this though, I feel like I don't have the experience to continue this, especially in the wrong place. I'm really just trying to familiarize myself with what can be done with the different jobs and our data, then will try to work out some more definite tasks from there. An upgrade to 8 soon will also help.
Okay so now that helps a lot if you would just give it a screenshot of the entire screen to start with and actually if you even did bigger I would be able to point exactly where to go find what you're looking for.
If you click on one of the anomalies and then open it up at the bottom, there is all the details including what the categorization pattern/ details are.
As a new member to the forum, I'm trying to help you understand. If you provide more details we can answer the question much quicker if you just give us a little tiny partial screen shot and only a few words we're going to have to ask a lot of questions.
You're doing great! I'm glad you're in there. Trying to figure this out. Just remember, bring a lot of context and you'll get answers quickly
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.
Maybe to be more clear: I don't know how they are being categorized.
. I am also fine dropping this though, I feel like I don't have the experience to continue this, especially in the wrong place. I'm really just trying to familiarize myself with what can be done with the different jobs and our data, then will try to work out some more definite tasks from there. An upgrade to 8 soon will also help.