Fields are not visible in machine learning job

[Sripal]

Hi All,

I am new to Machine learning jobs in kibana.

I've tried to create a single metric job using my data available in elasticsearch.
But my field are not visible in filters Aggregation.

Except for the aggregation type distinct_count .

By using "distinct_count" am unable to do any prediction. Please help me.
The fields am trying to use for the prediction is in NUMBER type.

Appreciate your response.

[Sripal]

[Sripal]

I'm using ELK 6.4.2
I tried to remove and install x-pack also.
But no luck anybody please help me. I'm doing anything wrong?

[Sripal]

Any response would appreciated.

[richcollier]

Hmm...I don't seem to have a problem with selecting distinct_count for a field that is numerical:

And here's the mapping of that field:

          "responsetime" : {
            "type" : "float"
          }

What is the name of the field you're interested in and what is its mapping?

[Peter_Harverson (Pete Harverson)]

@Sripal could you also check that the number type fields you want to use in your Single Metric job are listed in the Kibana Index Patterns tab, when filtering for number (switch to the Kibana Management tab, then Index Patterns, select the index pattern you are using in your job, and then filter for number) .

For example, in this index pattern, there are no aggregatable number type fields available for use in an ML job:

Thanks
Pete

[Sripal]

Thankyou for the response @Peter_harverson & @richcollier.

Please review my index pattern screenshot all fields treated as string.

my config file

filter {
grok {
match => {
"message" => '%{NOTSPACE:clientip}:- %{NOTSPACE:user} %{NOTSPACE:pass} %{NOTSPACE:role} [%{HTTPDATE:timestamp}] "%{WORD:method} %{DATA:request} HTTP/%{NUMBER:httpversion}" %{NUMBER:responsecode} %{NUMBER:response} %{NOTSPACE:id1} %{NOTSPACE:id2} %{NOTSPACE:id3} %{NOTSPACE:id4} %{NOTSPACE:id5}'
}
}

}

Please let me know my mistake .

[Sripal]

@Peter_Harverson I filtered my index pattern with NUMBER please find the below screenshot

[Sripal]

can anybody help me how to typecast the field from string to integer.

[Sripal]

I've tried the below

[richcollier]

Hello - it seems as if your mappings are a bit of a mess. You should not have everything be type string. Time stamps should be type date and if you have fields named xxxxx.keyword then those fields should be of type keyword.

Looks like you need to re-index your data with proper mappings. A great blog with good info: https://www.elastic.co/blog/a-practical-introduction-to-logstash

[Sripal]

@richcollier Thanks much. I didn't mutate my fields. Now i can able to see the numeric type fields.

But i'm unable to do forecast my job.

[richcollier]

I think this problem is related to the other problem you're having as described here: Datafeed not happening in ml job

[Sripal]

@richcollier Thanks for the response.