Anomaly Detection for input logs (Elastic Agents)

I want to create something like a Machine Learning or Threshold rule that triggers when my infrastructure loses events from elastic agents.

For example, the medium of events per day is 500k logs. But if I will get only 100k for the day, I would like to get a notification/alert that informs me about losing events so I may immediately take a look at what is going on.

I tried to create a new job in Anomaly Detection, but it looks like there is no option to create a rule that works in real-time, not based on a dedicated time range.

My Elastic version: v 8.8.2.


Machine Learning jobs are meant for realtime while setting up the job it will ask for a historical time frame to learn from (you can say none / very little).. but a the end, you can tell it to run ongoing and it will alert you on as anomalies going forward.

BTW the log rate analysis job does exactly what you describe.
This is a bit of a wizard...

On a normal Job after you create the job you it runs on the data set you give it (what are calling "not based on a dedicated time range.")

After it is created then ou go to the job and start it ongoing...

SOme Docs

Hi Stephen! I tried to start with Elastic Observability Anomalies, but I got the error and there are no details I should look for.

When I open the Log rate tab, Elastic loads indexes for a few minutes. Then I see this error and can't create an ML job. The button is unclickable because of this issue.

Can you please suggest?

I found an error code here (raw 8430):

"xpack.infra.analysisSetup.indicesSelectionNetworkError": "We couldn't load your index configuration"

What Type of Data are you collecting?
How are you collecting the data?
What Index Pattern is your logs in?

  1. What Type of Data are you collecting?
  • events from Windows machines
  1. How are you collecting the data?
  • using elastic agents
  1. What Index Pattern is your logs in?
  • logs-*

I wonder what is that kibana_sample_data_logs* index. I may get an error because of this index. Is it possible that the index exists in some config file, but in fact it doesn't exist in Kibana? And when Kibana is trying to load that index - Kibana can't find it and returns an error.

The kibana is sample data that You or someone installed. it can be removed. There is even a button to remove it? If you go to the home screen then add data and then I think it's under sample data then there should be an option to remove it. Or just go to Data views and remove the data view

Is this elastic cloud or a self-managed cluster? And do you have a machine learning node?

We have deleted the kibana_sample index as you said, but we get the same error.

When I try to create a ML setup for log rate detection, the Elastic loads the indexes for some minutes:

Then I get the same error as before:

Answers to your questions:
Elastic is a self-managed cluster. We have a machine learning node.

Yes but have you deleted the Data View?

Question Machine Learning is a licensed feature, do you have a license? If so you can open a support ticket.

We are licensed and the dataview was deleted.

Can you suggest how I can open a support ticket?

Whoever procured you licenses should have been contacted about setting up support contact.

This is support portal


Thank you Stephen for your hard work. You helped me a lot, really.
I started a Case with Elastic Support. I hope this issue will be fixed.

Thank you again!

1 Like

Rather than start a whole new topic, I figure I will ask here since it is mildly related: I have gotten the ML categorization working with our (many) log files, but have not found a way to identify what the actual category fields are outside of "mlcategory#". Would this be due to still using 7.17 or am I just missing something very obvious? I felt like there were more discovery options on the trial with 8. Sorry for the basic question, I am pretty new to this and trying to catch up as fast as I can.

Hi @Anomalous_User Welcome to the community!.

Actually you probably should have opened a new topic since this one is marked to solved it. Not many people will look at it.

There's been quite a bit of improvement in 8.x particularly around log categorization and spike and dip analysis.

I'm not quite sure what you're asking perhaps you could give an example.

Hi, thanks for the response. I am trying to determine the fields associated with the different categories, but don't know where to start... I can look at examples, but have no hint what fields it is alerting on.
image Maybe to be more clear: I don't know how they are being categorized.

Ok those are influencers ... again not sure what you are doing/asking, not sure what kind of job that is etc...

I would suggest opening a new topic and go through what you are trying to accomplish and understand and what you have set up step by step...

I am trying to setup an XYZ Job so that I can anomalies in my ABC data etc...

Here are the steps I have followed... can you help my understand ...

you are going to need to provide much more context if you want help

Thanks, I'm really just trying to get a feel for it with our data, understand how it is categorizing, and figure out how it can help me. If I can only understand the different categories (influencers) as "mlcategory_" without knowing how they are being categorized, it leaves me a bit lost. Thanks for the help, I'll keep on reading.

Ok so even simple statement like this you will need to clarify...

Apologies but I am not trying to be difficult but there are so many topics / details with ML and / or Categorization we won't be able to help unless you start at the beginning... You said you are just getting started, I am just trying to help you get and answer...

What data, What Categorization there are many ways to set up categorization ... are you talking about logs? one of the OOTB Jobs? if it is logs with message fields the categorization is based on pattern matching...

Here are few examples....

I was already responding in the wrong place and didn't want to draw this out, thank you for your commitment to solving this.

I am limited to one picture per post as a new user.

I created the job through Observability > Logs > Anomalies > Create Job > Categorization

Then used anomaly explorer:

This was in an attempt to find out how the different categorization is being determined, so I can find out what is actually going on behind anomalous behaviors associated with the many different "mlcategory_"

Seriously, thank you for the help :heart:. I am also fine dropping this though, I feel like I don't have the experience to continue this, especially in the wrong place. I'm really just trying to familiarize myself with what can be done with the different jobs and our data, then will try to work out some more definite tasks from there. An upgrade to 8 soon will also help.

Okay so now that helps a lot if you would just give it a screenshot of the entire screen to start with and actually if you even did bigger I would be able to point exactly where to go find what you're looking for.

If you click on one of the anomalies and then open it up at the bottom, there is all the details including what the categorization pattern/ details are.

As a new member to the forum, I'm trying to help you understand. If you provide more details we can answer the question much quicker if you just give us a little tiny partial screen shot and only a few words we're going to have to ask a lot of questions.

You're doing great! I'm glad you're in there. Trying to figure this out. Just remember, bring a lot of context and you'll get answers quickly