Datafeed not happening in ml job

[Sripal]

Hi all,

I'm new to ml job.

I've created the ml job in kibana. I got an error

1. My data with timestamp (It's a sample data of one log file ) all data are feed at one shot.

2. While validating my job i got the below warnings.

3. In job management tab i got only one proceed data for my job.

4. My job config

5. JSON file

Can anybody help me.

Appreciate your responses

[Sripal]

I have 100 thousand documents in elastic search but it is processed only 1.

please let me know my mistake.

My expectation in ml job is:

1. filed "response" is the response time of the HTTP request.
   I want to find the mean of "response" and to predict the mean value in future.

2. I tried SUM also but, i got same response.

Anyone guide me how to do ml job.

[richcollier]

It looks like the main crux of your problem is that although you have 100k+ documents in Elasticsearch, they all have the same timestamp. In logstash, you must have not properly extracted the timestamp from the logs and used that to write the @timestamp field (??) See the "Date filter" section of: https://www.elastic.co/blog/a-practical-introduction-to-logstash

The ML job needs to see documents over long-ish periods of time (hours, days, weeks, etc.). Until you fix the timestamps, your ML jobs will never work.

[Sripal]

Hi @richcollier , Thank you for the response. Please review the below screenshot of my log line

Here ;;

1. I tried to read the sample log file.
2. @timestamp is taken as when the log lines are read and sent to elastic search( Dec 19 2018 xxxx). but the actual log time of log in the file is (Dec 4 2018 xxx)
3. currently the field timestamp (which i defined) in text type. I can mutate it to date.
4. How to i get the actual log time (Dec 4 2018 xxx) in elasticsearch???

could you please guide me how to achieve this.

[Sripal]

@richcollier , I'll try this suggestion and let you know the results.

[Sripal]

@richcollier Thanks much.

Issue is resolved.

I mapped my logtime to the timestamp.

filter {
grok {
match => {
"message" => '%{NOTSPACE:clientip}:- %{NOTSPACE:user} %{NOTSPACE:pass} %{NOTSPACE:role} [%{HTTPDATE:logtime}] "%{WORD:method} %{DATA:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:responsecode} %{NUMBER:response} %{NOTSPACE:id1} %{NOTSPACE:id2} %{NOTSPACE:id3} %{NOTSPACE:id4} %{NOTSPACE:id5}'
}
}

date {
match => [ "logtime", "dd'/'MMM'/'yyyy:HH:mm:ss Z" ]

	}

mutate {
convert => {
"responsecode" => "integer"
"response" => "integer"
}
}
}

Now its working perfectly