Need review for my REST query (template modification)

Elastic Stack Elasticsearch
1

Hi!

I am using the usual ELK stack with the default template (
http://pastebin.com/DtYiazVr
http://www.google.com/url?q=http%3A%2F%2Fpastebin.com%2FDtYiazVr&sa=D&sntz=1&usg=AFQjCNEIrorp9e9dSh98HTjQoPQoEor3Vg).
In every log message, the date in stored in field named "log_date", which
the date filter converts in a "@timestamp". I want to set the "log_date"
field as "not_analyzed" so that I can sort it in Kibana without getting
weird results.

I built the following query

curl -XPUT localhost:9200/_template/template_1 -d '
{
"template" : "logstash-*",
"properties" : {
"log_date" : {
"type" : "string",
"index" : "not_analyzed"
}
}
}

Can you confirm that the request is correct? I have doubts about the
template name. I thought about calling url
localhost:9200/_template/logstash in order to modify the existing template
rather than creating a new one, but I am afraid of what could happen the
day I restart logstash, so my thinking is that if all works as intended,
both logstash default template and that one will apply.

Also I believe that templates are only about future data. Is there any way
to retro apply it back on existing indexes?

Thanks for your help

Aldian

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/1c7fe34e-7792-409d-83ab-3c39fe883e95%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

Don't tell me nobody here ever made such a simple request?

On Thursday, January 22, 2015 at 11:57:26 AM UTC+1, Aldian wrote:

Hi!

I am using the usual ELK stack with the default template (
http://pastebin.com/DtYiazVr
http://www.google.com/url?q=http%3A%2F%2Fpastebin.com%2FDtYiazVr&sa=D&sntz=1&usg=AFQjCNEIrorp9e9dSh98HTjQoPQoEor3Vg).
In every log message, the date in stored in field named "log_date", which
the date filter converts in a "@timestamp". I want to set the "log_date"
field as "not_analyzed" so that I can sort it in Kibana without getting
weird results.

I built the following query

curl -XPUT localhost:9200/_template/template_1 -d '
{
"template" : "logstash-*",
"properties" : {
"log_date" : {
"type" : "string",
"index" : "not_analyzed"
}
}
}

Can you confirm that the request is correct? I have doubts about the
template name. I thought about calling url
localhost:9200/_template/logstash in order to modify the existing template
rather than creating a new one, but I am afraid of what could happen the
day I restart logstash, so my thinking is that if all works as intended,
both logstash default template and that one will apply.

Also I believe that templates are only about future data. Is there any way
to retro apply it back on existing indexes?

Thanks for your help

Aldian

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/2de42363-b3a8-4561-bda8-b844d4c287a4%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

3

On Thursday, January 22, 2015 at 11:57 CET,
Aldian aldian.gp@gmail.com wrote:

I am using the usual ELK stack with the default template
([1]http://pastebin.com/DtYiazVr). In every log message, the date in
stored in field named "log_date", which the date filter converts in a
"@timestamp". I want to set the "log_date" field as "not_analyzed" so
that I can sort it in Kibana without getting weird results.

You're storing the same timestamp in two fields? Why?

I built the following query
curl -XPUT localhost:9200/_template/template_1 -d '
{
"template" : "logstash-*",
"properties" : {
"log_date" : {
"type" : "string",
"index" : "not_analyzed"
}
}
}
Can you confirm that the request is correct?

It looks okay. You may want to use the 'order' key to make sure these
two matching index templates are applied in a well-defined and obvious
order.

I have doubts about the template name. I thought about calling url
localhost:9200/_template/logstash in order to modify the existing
template rather than creating a new one, but I am afraid of what could
happen the day I restart logstash, so my thinking is that if all works
as intended, both logstash default template and that one will apply.

I disable Logstash's index template handling and maintain my own
template (that started out as a copy of Logstash's).

Also I believe that templates are only about future data. Is there any
way to retro apply it back on existing indexes?

You'll have to reindex the data, e.g. using es-reindex.

--
Magnus Bäck | Software Engineer, Development Tools
magnus.back@sonymobile.com | Sony Mobile Communications

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/20150129122021.GE30366%40seldlx20533.corpusers.net.
For more options, visit https://groups.google.com/d/optout.

4

Thank you for replying, I am going to give it a try.

As for your question, the way Kibana displays timestamps is a bit cryptic.
The content of the log_date field is way more readable by a human being.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/9553d7cf-584b-4e35-a148-332e30934d7d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

5

Hi

I tried the request today and got back an "{"acknowledged":true}". Then I
tried

curl -XGET localhost:9200/_template/logstashdate
{"logstashdate":{"order":1,"template":"logstash-*","settings":{},"mappings":{},"aliases":{}}}

As you can see the properties have been completely ignored. I suspect I
should have included those into the mapping field, something like what is
done with the default template. Next I am planning to try this one:

curl -XPUT localhost:9200/_template/logstashdate -d '
{
"order": 1,
"template" : "logstash-*",
"mappings": {
"default": {
"properties" : {
"log_date" : {
"type" : "string",
"index" : "not_analyzed"
}
}
}
}
}

I am mostly guessing there, as the documentation is mostly example based
and I did not find the relevant one.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/49f241d0-131e-4c4f-b78e-ca17f55878d9%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.