Anomaly Detection Jobs - Various warning messages

Elastic Stack Elasticsearch
1

We are seeing various warning message below. Is there a guide to kickstart how we can kick start our troubleshooting?

Warning

  • datafeed is encountering errors submitting data for analysis
  • exception while flushing jobs

Warning

  • datafeed is encountering errors extracting data
  • 1 remote clusters out of 2 were skipped when peforming datafeed search

Warning

  • Datafeed has missed xxxx documents due to ingest latency
  • consider increasing query_delay and investigate the cause of high latency in your ingestion process
2

Hello @Whoami1980

It depends upon the error as to where we should start reviewing the issue.

We do have above elastic documentation where common issues are captured.

Or check for similar errors over here like below example :

Thanks!!

3

@Tortoise

For errors submitting data for analysis and error extracting error.
I was digging into the Elastic troubleshoot docs but had no clue where i should start with

For errors missed xxxx documents due to ingest latency
I read through Sophie reply in that post. In our cluster we have 3 roles having the ingest roles and 2 nodes having the ml roles. Do u mean we should have another node for ingest roles?

Kindly advice

4

@Tortoise

i found these with some able to give me some basic info . but still dont know how to troubleshoot the errors in particular.

GET _ml/datafeeds/

GET _xpack/ml/datafeeds/

"error": "no handler found for uri [/_xpack/ml/datafeeds/?pretty=true] and method [GET]"

GET /_ml/anomaly_detectors/
GET /_ml/anomaly_detectors/{job_id}/_stats
GET /_ml/anomaly_detectors/elastic_cluster_cpu_load/_stats