How to make timeline from Hadoop data?

Elastic Stack Elasticsearch
1

Hello!
I'm currently using Elasticsearch Hadoop from my Spark Structured Streaming job which generates data having the following schema:
root
|-- id: string (nullable = true)
|-- eventTime: timestamp (nullable = true)

I get all my data into Elasticsearch and can discover it. But When I try to use Machine Learning with my index, I always get the error "Index pattern spark-index* is not time based". But I do have a time field.
My index is created with this command:
PUT spark-index?pretty
{
"settings" : {
"number_of_shards" : 1
},
"mappings" : {
"exc" : {
"properties" : {
"id" : { "type" : "text" },
"eventTime":{ "type" : "date"
}
}
}
}
}
What I am doing wrong here?
Hope my first question is not too stupid!

2

When you added the index pattern, did you make sure you defined it as time based by picking a time field?

3

What exactly do you mean by defining it time based? How do I check this?
I defined the mapping of field eventTime as type date.

4

As per https://www.elastic.co/guide/en/kibana/current/tutorial-define-index.html

5

After carefully reading the linked article, especially the "Note" paragraph I got it to work.
The trick was to first generate an index with my field named "eventTime" having the type "date". Then importing the some data and after that generating an index pattern.
Thanks for the hint!

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.