Elasticsearch not detecting timestamp field from HBase table

Elastic Stack Elasticsearch
1

Hi,

I am loading my data from an Hbase table into Elasticsearch engine using
Pig.

I have a column in my Hbase table called timestamp (with data like
"2014-03-18 11:05:37.503").
I want this field to be automatically detected as the timestamp field by
the Elasticsearch (I need to add time filter on my Kibana dashboard).

Here's what I do in the PIG grunt shell:

REGISTER /path-to-my/elasticsearch-hadoop.jar;
A = LOAD 'hbase://demo' USING
org.apache.pig.backend.hadoop.hbase.HBaseStorage('cf:timestamp') as (
timestamp:chararray);
STORE A INTO 'demo/demo' USING org.elasticsearch.hadoop.pig.ESStorage();

Should I store the values for the field in any particular format?

How do I alter the above steps to suit my requirement?

Thanks.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/93b5f866-464e-4457-917e-39a5ca1ba65f%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

Hi,

You can either convert the timestamp field in Pig to UTF format (try the dateTime type and its associated functions [1])
or you can create the index mapping in Elasticsearch before indexing and define the time field format accordingly [2]

[1] Built In Functions
[2] Elasticsearch Platform — Find real-time answers at scale | Elastic

On 3/18/14 9:26 AM, sandip002@gmail.com wrote:

Hi,

I am loading my data from an Hbase table into Elasticsearch engine using Pig.

I have a column in my Hbase table called timestamp (with data like "2014-03-18 11:05:37.503").
I want this field to be automatically detected as the timestamp field by the Elasticsearch (I need to add time filter on
my Kibana dashboard).

Here's what I do in the PIG grunt shell:

REGISTER /path-to-my/elasticsearch-hadoop.jar;
A = LOAD 'hbase://demo' USING org.apache.pig.backend.hadoop.hbase.HBaseStorage('cf:timestamp') as ( timestamp:chararray);
STORE A INTO 'demo/demo' USING org.elasticsearch.hadoop.pig.ESStorage();

Should I store the values for the field in any particular format?

How do I alter the above steps to suit my requirement?

Thanks.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to
elasticsearch+unsubscribe@googlegroups.com mailto:elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/93b5f866-464e-4457-917e-39a5ca1ba65f%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/93b5f866-464e-4457-917e-39a5ca1ba65f%40googlegroups.com?utm_medium=email&utm_source=footer.
For more options, visit https://groups.google.com/d/optout.

--
Costin

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5327FE37.5030200%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

3

To clarify - by UTF, I meant ISO format :slight_smile:

On 3/18/14 10:05 AM, Costin Leau wrote:

Hi,

You can either convert the timestamp field in Pig to UTF format (try the dateTime type and its associated functions [1])
or you can create the index mapping in Elasticsearch before indexing and define the time field format accordingly [2]

[1] Built In Functions
[2] Elasticsearch Platform — Find real-time answers at scale | Elastic

On 3/18/14 9:26 AM, sandip002@gmail.com wrote:

Hi,

I am loading my data from an Hbase table into Elasticsearch engine using Pig.

I have a column in my Hbase table called timestamp (with data like "2014-03-18 11:05:37.503").
I want this field to be automatically detected as the timestamp field by the Elasticsearch (I need to add time filter on
my Kibana dashboard).

Here's what I do in the PIG grunt shell:

REGISTER /path-to-my/elasticsearch-hadoop.jar;
A = LOAD 'hbase://demo' USING org.apache.pig.backend.hadoop.hbase.HBaseStorage('cf:timestamp') as ( timestamp:chararray);
STORE A INTO 'demo/demo' USING org.elasticsearch.hadoop.pig.ESStorage();

Should I store the values for the field in any particular format?

How do I alter the above steps to suit my requirement?

Thanks.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to
elasticsearch+unsubscribe@googlegroups.com mailto:elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/93b5f866-464e-4457-917e-39a5ca1ba65f%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/93b5f866-464e-4457-917e-39a5ca1ba65f%40googlegroups.com?utm_medium=email&utm_source=footer.

For more options, visit https://groups.google.com/d/optout.

--
Costin

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/5327FF0D.4010001%40gmail.com.
For more options, visit https://groups.google.com/d/optout.