ML Datafeed lookback retrieved no data

TheHunter1 · January 4, 2021, 4:07pm

I am trying to run the integrated jobs for winlogbeat data, and most of them are having an error Datafeed lookback retrieved no data.

I already had the same problem using Elasticsearch version 8.0.0 from code source, So I decided to install the version 7.10.1, and even the beats are 7.10.1, and still have the same error.

So to test, I tried to clone one of the jobs, but just clicking on clone I get an error

[request body.earliestMs]: expected value of type [number] but got [null]

and when I continue till the validation, I am getting an error:

Job Validation Error

[request body.duration.start]: expected value of type [number] but got [null]

knowing that I am using a dedicated node for Machine learning by configuring the node as bellow:

node.roles: [ ml ]

Could you please help me to figure out why I am getting this error ?
Thanks

droberts195 · January 5, 2021, 1:05pm

[request body.duration.start]: expected value of type [number] but got [null] is due to a bug in the UI that was fixed in https://github.com/elastic/kibana/pull/86114. That fix will be in 7.11.

But both that and the original error of Datafeed lookback retrieved no data suggest that the datafeeds for your Winlogbeat jobs have ended up configured with searches that are not finding your Winlogbeat data. If you expand one of the rows on your jobs list you'll be able to see the datafeed config, including the indices that it's searching and the query it's using. Check whether you would expect a search using them to find your Winlogbeat data.

TheHunter1 · January 5, 2021, 2:11pm

Thanks for your answer @droberts195,

Here is an example of the datafeed configuration of one of the jobs that are not working, and as it's shown in the picture bellow, it is searching data in the right indice winlogbeat-*,

When I try:

GET _search
{
  "query": {
    "bool":{
      "filter":[{
        "term":{
          "event.action":"Process Create (rule: ProcessCreate)"
        }
      },
      {
        "term":{
          "agent.type":"winlogbeat"
        }
      }
      ]}
  }
}

I am getting no result too:

#! Deprecation: this request accesses system indices: [.apm-agent-configuration, .apm-custom-link, .async-search, .kibana_1, .kibana_task_manager_1, .ml-config, .security-7, .tasks, .transform-internal-005, .triggered_watches, .watches], but in a future major version, direct access to system indices will be prevented by default
{
  "took" : 2215,
  "timed_out" : false,
  "_shards" : {
    "total" : 36,
    "successful" : 36,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 0,
      "relation" : "eq"
    },
    "max_score" : null,
    "hits" : [ ]
  }
}

Is that mean that I don't have enough data ?

droberts195 · January 5, 2021, 2:33pm

Yes, try broadening the search to find out which of the two terms you don't have:

GET winlogbeat-*/_search
{
  "query": {
    "bool":{
      "filter":[{
        "term":{
          "event.action":"Process Create (rule: ProcessCreate)"
        }
      }
      ]}
  }
}

and:

GET winlogbeat-*/_search
{
  "query": {
    "bool":{
      "filter":[{
        "term":{
          "agent.type":"winlogbeat"
        }
      }
      ]}
  }
}

Do either of those find anything?

TheHunter1 · January 5, 2021, 2:49pm

When I run the first query I am getting no result :

{
  "took" : 0,
  "timed_out" : false,
  "_shards" : {
    "total" : 1,
    "successful" : 1,
    "skipped" : 0,
    "failed" : 0
  },
  "hits" : {
    "total" : {
      "value" : 0,
      "relation" : "eq"
    },
    "max_score" : null,
    "hits" : [ ]
  }
}

And when I run the second query I get results.

I will let the winlogbeat to turn more in machins and then try again

Thanks for your help @droberts195

droberts195 · January 5, 2021, 3:14pm

OK it might be worth having a browse through your winlogbeat-* indices using the Discover app. Do you see events that contain something similar to, but not identical to Process Create (rule: ProcessCreate)? I am wondering whether that string varies a bit between different versions of Windows, or whether it gets localized. For example, do you see documents in winlogbeat-* that have Process Create translated into some other language?

You might have discovered a flaw where these jobs have been accidentally tailored to one specific version of Windows or to Windows running in the US English locale only or something like that.

TheHunter1 · January 5, 2021, 3:58pm

I used discover app to browse into my winlogbeat-* indice and then Toggle the column event.Action to see what I am getting inside.
there was some result which are translated to French language, and some are in English, but I didn't find in both languages Process Create (rule: ProcessCreate)

here is a list of all what I get in event.Action field:

Événements d’arrêts inattendus d’applications
Ouvrir la session
group-membership-enumerated
Modification de l’état de la sécurité
Défragmentation en ligne
Général
Cliché instantané
Enregistrement/récupération
Modification de la stratégie d’audit
kerberos-service-ticket-renewed
kerberos-authentication-ticket-requested
Directory Service Access
user-member-enumerated
kerberos-service-ticket-requested
User Account Management
gent de mise à jour automatique Windows Update
logon-failed
General
Guest Library API
logged-out
credential-validated
logged-in-special
logged-in-explicit
logged-in

droberts195 · January 5, 2021, 4:27pm

A bit of searching led me to https://github.com/elastic/kibana/issues/50003:

Therein lies the problem. It is not obvious at first, but jobs like "windows_anomalous_process_all_hosts_ecs" actually depend on the field event.action being a sysmon value "ProcessCreate" as opposed to the expected "created-process" that windows event id's give.

So it looks like what you are seeing is a problem that has been reported before but nothing has been done about it yet.

It seems that you have to enable Sysmon for most of the Winlogbeat jobs to work.

TheHunter1 · January 5, 2021, 4:49pm

So I have just tried to install sysmon in one machine by running the command .\Sysmon.exe -i -n -accepteula and now I am seeing Process Create (rule: ProcessCreate) in the event.action field.

Tomorrow morning I will install sysmon in all my windows servers, and then I will try again to run the machine Learning jobs and I will keep you updated of the results

Thanks for your answers

TheHunter1 · January 6, 2021, 2:22pm

Hello again,

So after Installing sysmon, most of the winlogbeat machine Learning jobs are working, (8 are working, and still have problem in 3 jobs)

the jobs which are not working now are:

windows_anomalous_script
windows_rare_metadata_process
windows_rare_metadata_user

The query of the windows_anomalous_script job is:

GET _search
{
  "query": {
    "bool":{
      "filter":[{
        "term":{
          "winlog.channel":"Microsoft-Windows-PowerShell/Operational"
        }
      }]
    }
  }
}

and for the 2 others jobs is:

GET _search
{
  "query": {
    	"bool":{
    	  "filter":[{
    	    "term":{
    	      "destination.ip":"169.254.169.254"
    	    }
    	  }]
    	}
  }
}

Could you please tell me why I am not getting results for the first job, and why in 2 the other jobs the query is searching just for that IP address ??

droberts195 · January 6, 2021, 2:49pm

This relates to PowerShell scripts. If you don't use PowerShell scripts it's not surprising you don't have any events containing this field.

169.254.169.254

This is the address of the metadata service that's available in Azure and AWS - see https://docs.microsoft.com/en-us/azure/virtual-machines/windows/instance-metadata-service and https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html. If your servers are not in Azure or AWS then, again, it's not surprising that you don't have events that contain this IP address.

I can see that we should try to make the user experience nicer when there is a set of ML jobs and it's expected that some of them won't have any input data and others will. I will take this up internally. For the time being I would suggest you just stop those jobs that aren't relevant to your environment so that they don't use up any resources. Just leave the 5 open that are relevant to you.

TheHunter1 · January 6, 2021, 2:50pm

Thanks a lot for all your answers and explanations , it means that everything is working fine