Elasticsearch field mapping, dynamic_templates

Elastic Stack Elasticsearch
1

As I did mention here
https://groups.google.com/forum/#!topic/elasticsearch/7Bn5Pc6TSgs
I have some sorting issue when date is stored in string format.

I did try to set field format as date, but it is overide when date is
inserted into index.The mapping was:

curl -XPUT localhost:9200/*/_mapping/loglog -d '
{
"loglog" : {
"properties" : {
"testdate7" : {"type" : "date", "format" : "yyyy-MM-dd
HH:mm:ss.SSSSSS", "store" : true }
}
}
}
'

but after while it become:

"properties" : {
"testdate7" : {
"type" : "string",
"norms" : {
"enabled" : false
},

*I guess it may be caused by *

"dynamic_templates" : [ {
"string_fields" : {
"mapping" : {
"type" : "multi_field",
"fields" : {
"raw" : {
"index" : "not_analyzed",
"ignore_above" : 256,
"type" : "string"
},
"{name}" : {
"index" : "analyzed",
"omit_norms" : true,
"type" : "string"
}
}
},
"match" : "*",
"match_mapping_type" : "string"
}
} ],

So the question os, how create mappint that would cover that issue.The
message that is insterted looks like:
"ANYTHING (pid: 23291, thread: 4131280592) *******] [ aa.xx:555: MSG(3)
2014-06-09 10:50:08.255111 ... "

I did try to set
"dynamic_date_formats" : ["date_optional_time", "yyyy-MM-dd
HH:mm:ss.SSSSSS"],
but I think I did not set it properly.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/bf31ea3f-292b-451b-aa14-0ef2f3632f44%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

Hey,

using your curl call you only set the mapping for existing indices, but not
for newly created ones. You may want to consider using index templates for
this specific field.

--Alex

On Mon, Jun 9, 2014 at 4:45 PM, sirkubax jakubxmuszynski@googlemail.com
wrote:

As I did mention here
Redirecting to Google Groups
I have some sorting issue when date is stored in string format.

I did try to set field format as date, but it is overide when date is
inserted into index.The mapping was:

curl -XPUT localhost:9200/*/_mapping/loglog -d '
{
"loglog" : {
"properties" : {
"testdate7" : {"type" : "date", "format" : "yyyy-MM-dd
HH:mm:ss.SSSSSS", "store" : true }
}
}
}
'

but after while it become:

"properties" : {
"testdate7" : {
"type" : "string",
"norms" : {
"enabled" : false
},

*I guess it may be caused by *

"dynamic_templates" : [ {
"string_fields" : {
"mapping" : {
"type" : "multi_field",
"fields" : {
"raw" : {
"index" : "not_analyzed",
"ignore_above" : 256,
"type" : "string"
},
"{name}" : {
"index" : "analyzed",
"omit_norms" : true,
"type" : "string"
}
}
},
"match" : "*",
"match_mapping_type" : "string"
}
} ],

So the question os, how create mappint that would cover that issue.The
message that is insterted looks like:
"ANYTHING (pid: 23291, thread: 4131280592) *******] [ aa.xx:555: MSG(3)
2014-06-09 10:50:08.255111 ... "

I did try to set
"dynamic_date_formats" : ["date_optional_time", "yyyy-MM-dd
HH:mm:ss.SSSSSS"],
but I think I did not set it properly.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/bf31ea3f-292b-451b-aa14-0ef2f3632f44%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/bf31ea3f-292b-451b-aa14-0ef2f3632f44%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGCwEM8cXo_Hx7-UszUWcWsE51k0q4AMKNamr-6Zq1%2BLAVMMzg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

3

Hi Alex, That's more or less what I did:

curl -XGET localhost:9200/_template?pretty > template_all

edit template_all

and put it back:
curl -XPUT localhost:9200/_template/* -d @template_all

  • By ES is 1.0.1, I've seen that there is major change in templates in ES
    1.2. Do you think my task could be achieved faster? I had to dump config,
    edit id, and put it back. I'd wish to upload only "testdate"
    dynamic_templates in one step. the file: *
    cat template_all
    {
    "template" : "logstash-",
    "settings" : {
    "index.analysis.analyzer.default.stopwords" : "none",
    "index.refresh_interval" : "5s",
    "index.analysis.analyzer.default.type" : "standard"
    },
    "mappings" : {
    "default" : {
    "dynamic_templates" : [
    { "testdate": {
    "match": "testdate    ",
    "mapping": {
    "type": "date",
    "format" : "yyyy-MM-dd HH:mm:ss.SSSSSS"
    }
    }
    },
    {
    "string_fields" : {
    "mapping" : {
    "type" : "multi_field",
    "fields" : {
    "raw" : {
    "index" : "not_analyzed",
    "ignore_above" : 256,
    "type" : "string"
    },
    "{name}" : {
    "index" : "analyzed",
    "omit_norms" : true,
    "type" : "string"
    }
    }
    },
    "match_mapping_type" : "string",
    "match" : "*"
    }
    } ],
    "properties" : {
    "geoip" : {
    "dynamic" : true,
    "path" : "full",
    "properties" : {
    "location" : {
    "type" : "geo_point"
    }
    },
    "type" : "object"
    },
    "@version" : {
    "index" : "not_analyzed",
    "type" : "string"
    }
    },
    "_all" : {
    "enabled" : true
    }
    }
    }
    }

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/917d10ec-63b3-4d94-858a-2f2deeeba604%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

4

I dod migrate to ES 1.3.1

I did try to do the same trick, but it's fail to PUT oryginal, just dumped
settings.
Any ideas?

curl -XGET localhost:9200/_template?pretty > template_all

curl -XPUT localhost:9200/_template/*?pretty -d @template_all

{

  • "error" : "ActionRequestValidationException[Validation Failed: 1:
    template is missing;]",*
  • "status" : 500*
    }

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/b0981737-8788-4e90-8f2d-e8afc345c1a3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

5

I did migrate to ES 1.3.1

I did try to do the same trick as before, but it's fail to PUT oryginal,
just dumped settings.
Any ideas?

curl -XGET localhost:9200/_template?pretty > template_all

curl -XPUT localhost:9200/_template/*?pretty -d @template_all

{

  • "error" : "ActionRequestValidationException[Validation Failed: 1:
    template is missing;]",*
  • "status" : 500*
    }

I did not see any bigger change in dynamic_templates recently so I guess it
should work just like before.
Can You replicate this issue on your cluster?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/82e516e4-6308-4676-b0f9-e213d0650986%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

6

Ok, got it:

I did it template by template
When you capture template (for example: logstash

curl -XGET localhost:9200/_template/logstash?pretty > template_logstash

you get:

cat template_logstash
{
"logstash" : {
"order" : 0,
"template" : "logstash-",
"settings" : {
"index.analysis.analyzer.default.stopwords" : "none",
"index.refresh_interval" : "5s",
"index.analysis.analyzer.default.type" : "standard"
},
"mappings" : {
"default" : {
"dynamic_templates" : [ {
"string_fields" : {
"mapping" : {
"type" : "multi_field",
"fields" : {
"raw" : {
"index" : "not_analyzed",
"ignore_above" : 256,
"type" : "string"
},
"{name}" : {
"index" : "analyzed",
"omit_norms" : true,
"type" : "string"
}
}
},
"match_mapping_type" : "string",
"match" : ""
}
} ],
"properties" : {
"geoip" : {
"dynamic" : true,
"path" : "full",
"properties" : {
"location" : {
"type" : "geo_point"
}
},
"type" : "object"
},
"@version" : {
"index" : "not_analyzed",
"type" : "string"
}
},
"_all" : {
"enabled" : true
}
}
}
}
}

Then you need to remove the 1st parrent in the "json tree":
{
"logstash" :
}

So you end up with:

{
"order" : 0,
"template" : "logstash-",
"settings" : {
"index.analysis.analyzer.default.stopwords" : "none",
"index.refresh_interval" : "5s",
"index.analysis.analyzer.default.type" : "standard"
},
"mappings" : {
"default" : {
"dynamic_templates" : [ {
"string_fields" : {
"mapping" : {
"type" : "multi_field",
"fields" : {
"raw" : {
"index" : "not_analyzed",
"ignore_above" : 256,
"type" : "string"
},
"{name}" : {
"index" : "analyzed",
"omit_norms" : true,
"type" : "string"
}
}
},
"match_mapping_type" : "string",
"match" : ""
}
} ],
"properties" : {
"geoip" : {
"dynamic" : true,
"path" : "full",
"properties" : {
"location" : {
"type" : "geo_point"
}
},
"type" : "object"
},
"@version" : {
"index" : "not_analyzed",
"type" : "string"
}
},
"_all" : {
"enabled" : true
}
}
}
}

That can be put back to logstash template like:
curl -XPUT localhost:9200/_template/ -d @template_logstash*

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/86cf401d-931d-4a63-bf6b-65b6b35122e3%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.