Meachine learning anomaly detection error: no node found

Elastic Stack Elasticsearch
1

Hello,

I am trying elastic machine learning, I tried to run default job for audibeat and I am getting this error in kibana logs :

server    log   [11:59:16.059] [error][data][elasticsearch] [status_exception]: No node found to start datafeed [datafeed-linux_network_connection_discovery], allocation explanation [cannot start datafeed [datafeed-linux_network_connection_discovery], because the job's [linux_network_connection_discovery] state is [failed] while state [opened] is required]

and this error in Elasticsearch logs

[2020-10-19T11:59:14,538][INFO ][o.e.x.m.j.p.a.AutodetectProcessManager] [VSELK_MASTER] Successfully set job state to [opened] for job [linux_network_configuration_discovery]
[2020-10-19T11:59:14,609][INFO ][o.e.x.m.j.p.a.AutodetectProcessManager] [VSELK_MASTER] Successfully set job state to [failed] for job [linux_network_configuration_discovery]
[2020-10-19T11:59:14,702][INFO ][o.e.x.m.j.p.a.AutodetectProcessManager] [VSELK_MASTER] Opening job [linux_network_connection_discovery]
[2020-10-19T11:59:14,705][INFO ][o.e.x.m.j.p.a.AutodetectProcessManager] [VSELK_MASTER] [linux_network_connection_discovery] Loading model snapshot [N/A], job latest_record_timestamp [N/A]
[2020-10-19T11:59:15,922][INFO ][o.e.x.m.p.AbstractNativeProcess] [VSELK_MASTER] [linux_network_connection_discovery] State output finished
[2020-10-19T11:59:15,922][INFO ][o.e.x.m.p.l.CppLogMessageHandler] [VSELK_MASTER] [linux_network_connection_discovery] [autodetect/22168] [CResourceMonitor.cc@74] Setting model memory limit to 10 MB
[2020-10-19T11:59:15,922][ERROR][o.e.x.m.p.l.CppLogMessageHandler] [VSELK_MASTER] [linux_network_connection_discovery] [autodetect/22168] [CFieldConfig.cc@184] Cannot specify both a fieldname clause and a field config file
[2020-10-19T11:59:15,922][FATAL][o.e.x.m.p.l.CppLogMessageHandler] [VSELK_MASTER] [linux_network_connection_discovery] [autodetect/22168] [Main.cc@175] Field config could not be interpreted
[2020-10-19T11:59:15,923][ERROR][o.e.x.m.p.AbstractNativeProcess] [VSELK_MASTER] [linux_network_connection_discovery] autodetect process stopped unexpectedly: Cannot specify both a fieldname clause and a field config file
Field config could not be interpreted

Could someone help please !

Thanks :slight_smile:

2

Cannot specify both a fieldname clause and a field config file strongly suggests that the ML native processes are from a different version than the Elasticsearch server that's trying to run them and as a result the arguments provided to and expected by the autodetect process are not matching up correctly.

Please can you double check the exact versions of Elasticsearch and the ML autodetect process on the node that the log you pasted came from:

$ESHOME/bin/elasticsearch --version
$ESHOME/modules/x-pack-ml/platform/linux-x86_64/bin/autodetect --version
3

I am using Elasticsearch from code source, so I am trying the version 8.0.0

when I run bin/elasticsearch -- version I get:

Version: 8.0.0

and when I run

bin/autodetect --version

I get:

Model State Version 34
Quantile State Version 3
autodetect (64 bit): Version based on 8.0.0
4

Oh, you are building from source on the master branch (at least that's what Version based on 8.0.0 implies).

We recently did a change that affected the arguments sent from Java to C++: https://github.com/elastic/elasticsearch/pull/63865

You will need to incorporate the corresponding C++ changes, https://github.com/elastic/ml-cpp/pull/1540, into your local build of the C++. Depending on what you changed this might be as simple as a git pull in your ml-cpp clone.

5

Thanks for you answers @droberts195, I will try that and keep you informed of the results

6

Hello,

I tried to make git pull in both elasticsearch and ml-cpp and it worked perfectly as I am not getting that errors anymore.
But when I try to create some test jobs ( prebuilt rules ), I got a message on kibana screen "No overall data found" as it's shown in the picture bellow

image
image824×347 32.3 KB

image
image1493×572 23.2 KB

Is that mean that I don't have enough data (as I have just 4 hours or data in my index), or it just means that no anomaly has been detected in my data.

Thanks

7

Both!

You certainly need to allow a lot more data (ideally many days) to be seen by ML. Even so, it is possible that even after a while, there might not be anything remarkably unusual occurring in the data. If you want to test how ML works, the ideal situation is to have several days or even weeks worth of historical data to learn on, and then contrive the situation that you'd like to see get detected.

1 Like
8

Thanks for your answer @richcollier, I will let at least 1month of data in my index to test it again.
Best regards

9

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.