Mapping definition help (copy_to, dates)

Jeff_Fogarty · November 18, 2014, 4:40am

I'm looking to analyze network utilization data supplied by our vendor.
One challenge is the observed data and time are in separate fields.

I like to combine the fields date_observed and cst_observed into a
date/time field. Is this possible?

Below defines that mapping I've come up with, I get a few errors with the
below

No handler for type [timestamp] declared on fields

my_ts

cst_observed

{ "mappings": {
"utilization": {
"dynamic": "false", "properties": {
"cst_observed": {"type": "timestamp", "copy_to": "my_ts"}
,"date_observed": {"type": "date", "copy_to": "my_ts"}
,"inbound_mbps": {"type": "float"}
,"inbound_percent": {"type": "float"}
,"ip_addr": {"type": "ip"}
,"outbound_mbps": {"type": "float"}
,"outbound_percent": {"type": "float"}
,"provider": {"type": "string"}
,"reporting_begin": {"type": "date"}
,"reporting_end": {"type": "date"}
,"speed": {"type": "float"}
,"my_ts":{"type": "timestamp", "format": "HH:mm:ss
mm/dd/yyyy"}
}
}
}
}'

Thanks,

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/0109138a-4f04-4f66-9ab5-2f7f85fc9a77%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

johtani · December 6, 2014, 3:04pm

Hi Jeff,

elasticsearch doesn’t have “timestamp” type. elasticsearch have only date type.

And you should use “transform” instead of “copy_to”.

Example mapping using transform:
{
"mappings": {
"utilization": {
"dynamic": "false",

    "transform" : {
        "script" : "if (!ctx._source['cst_observed'].empty && !ctx._source['date_observed'].empty) ctx._source['my_ts'] = ctx._source['cst_observed'] + ' '+ ctx._source['date_observed']",
        "lang": "groovy"
    },
        "properties": {
            "cst_observed": {"type": "string"}
            ,"date_observed": {"type": "string"}
            ,"inbound_mbps": {"type": "float"}
            ,"inbound_percent": {"type": "float"}
            ,"ip_addr": {"type": "ip"}
            ,"outbound_mbps": {"type": "float"}
            ,"outbound_percent": {"type": "float"}
            ,"provider": {"type": "string"}
            ,"reporting_begin": {"type": "date"}
            ,"reporting_end": {"type": "date"}
            ,"speed": {"type": "float"}
            ,"my_ts":{"type": "date", "format": "HH:mm:ss mm/dd/yyyy"}
  }
}

}
}

That is only example.
I hope that those help you out.

Jun Ohtani
johtani@gmail.com
blog : http://blog.johtani.info
twitter : http://twitter.com/johtani

2014/11/18 13:40、Jeff Fogarty j.007ba7@gmail.com のメール：

I'm looking to analyze network utilization data supplied by our vendor. One challenge is the observed data and time are in separate fields.

I like to combine the fields date_observed and cst_observed into a date/time field. Is this possible?

Below defines that mapping I've come up with, I get a few errors with the below

No handler for type [timestamp] declared on fields
my_ts
cst_observed

{ "mappings": {
"utilization": {
"dynamic": "false", "properties": {
"cst_observed": {"type": "timestamp", "copy_to": "my_ts"}
,"date_observed": {"type": "date", "copy_to": "my_ts"}
,"inbound_mbps": {"type": "float"}
,"inbound_percent": {"type": "float"}
,"ip_addr": {"type": "ip"}
,"outbound_mbps": {"type": "float"}
,"outbound_percent": {"type": "float"}
,"provider": {"type": "string"}
,"reporting_begin": {"type": "date"}
,"reporting_end": {"type": "date"}
,"speed": {"type": "float"}
,"my_ts":{"type": "timestamp", "format": "HH:mm:ss mm/dd/yyyy"}
}
}
}
}'

Thanks,

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/0109138a-4f04-4f66-9ab5-2f7f85fc9a77%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/B58F56BA-643C-4B75-98E3-35DB5282FE73%40gmail.com.
For more options, visit https://groups.google.com/d/optout.

nik9000 · December 6, 2014, 3:24pm

Our you can always transform in you client application. The advantage of
transform is that it is done post source like copy_to. Meaning is you
like the original format for disk space and highlighting purposes you
should use transform. If you don't, transform in your app.

Nik

On Sat, Dec 6, 2014 at 10:04 AM, Jun Ohtani johtani@gmail.com wrote:

Hi Jeff,

elasticsearch doesn’t have “timestamp” type. elasticsearch have only date
type.

Elasticsearch Platform — Find real-time answers at scale | Elastic

And you should use “transform” instead of “copy_to”.

Elasticsearch Platform — Find real-time answers at scale | Elastic

Example mapping using transform:
{
"mappings": {
"utilization": {
"dynamic": "false",
    "transform" : {
        "script" : "if (!ctx._source['cst_observed'].empty &&
!ctx._source['date_observed'].empty) ctx._source['my_ts'] =
ctx._source['cst_observed'] + ' '+ ctx._source['date_observed']",
"lang": "groovy"
},
"properties": {
"cst_observed": {"type": "string"}
,"date_observed": {"type": "string"}
,"inbound_mbps": {"type": "float"}
,"inbound_percent": {"type": "float"}
,"ip_addr": {"type": "ip"}
,"outbound_mbps": {"type": "float"}
,"outbound_percent": {"type": "float"}
,"provider": {"type": "string"}
,"reporting_begin": {"type": "date"}
,"reporting_end": {"type": "date"}
,"speed": {"type": "float"}
,"my_ts":{"type": "date", "format": "HH:mm:ss mm/dd/yyyy"}
}
}
}
}

That is only example.
I hope that those help you out.

Jun Ohtani
johtani@gmail.com
blog : http://blog.johtani.info
twitter : http://twitter.com/johtani

2014/11/18 13:40、Jeff Fogarty j.007ba7@gmail.com のメール：

I'm looking to analyze network utilization data supplied by our vendor.
One challenge is the observed data and time are in separate fields.

I like to combine the fields date_observed and cst_observed into a
date/time field. Is this possible?

Below defines that mapping I've come up with, I get a few errors with
the below

No handler for type [timestamp] declared on fields
my_ts
cst_observed

{ "mappings": {
"utilization": {
"dynamic": "false", "properties": {
"cst_observed": {"type": "timestamp", "copy_to": "my_ts"}
,"date_observed": {"type": "date", "copy_to": "my_ts"}
,"inbound_mbps": {"type": "float"}
,"inbound_percent": {"type": "float"}
,"ip_addr": {"type": "ip"}
,"outbound_mbps": {"type": "float"}
,"outbound_percent": {"type": "float"}
,"provider": {"type": "string"}
,"reporting_begin": {"type": "date"}
,"reporting_end": {"type": "date"}
,"speed": {"type": "float"}
,"my_ts":{"type": "timestamp", "format": "HH:mm:ss
mm/dd/yyyy"}
}
}
}
}'

Thanks,

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/0109138a-4f04-4f66-9ab5-2f7f85fc9a77%40googlegroups.com
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/B58F56BA-643C-4B75-98E3-35DB5282FE73%40gmail.com
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAPmjWd2AY%2BKGoKxBFPSsayrGJ12VhLekFctmAvdCZo5SB95SdA%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.