Mapping created using Template does not work

Elastic Stack Elasticsearch
1

Hi,
I have setup ELK stack and I am going by default index name, which is
logstash-YYYY.MM.DD . Since this is the only index format I have, I decided
to create a template file, so that whenever new index gets created i can
set up the mapping property. I am not able to push the data to
elasticsearch if my index mapping gets created from template. May I know
where am I wrong ?

Here is my mapping file content:
{
"X_Server" : {
"properties" : {
"@timestamp" : {
"type" : "date",
"format" : "dateOptionalTime"
},
"@version" : { "type" : "string" },
"class" : { "type" : "string" },
"file" : { "type" : "string"},
"message": {"type": "string"},
"host" : { "type" : "string", "index": "not_analyzed" }
}}}

My template file content is

{
"template": "logstash-*",
"settings" : {
"index.number_of_shards" : 3,
"index.number_of_replicas" : 1,
"index.query.default_field" : "@message",
"index.routing.allocation.total_shards_per_node" : 2,
"index.auto_expand_replicas": false
},
"mappings": {
"X_Server": {
"_all": { "enabled": false },
"_source": { "compress": false },
"properties" : {
"class" : { "type" : "string", },
"host" : { "type" : "string", "index" : "not_analyzed" },
"file" : { "type" : "string" },
"message" : { "type": "string"}
}
}}}

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/b1d382b5-0fa7-4a2c-96f0-150d856482cc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

2

Hey,

can you include a full example including data you have been indexing? Or
include error messages, as you wrote you cannot index any data anymore?

Also, your JSON is not valid as the mapping for the class field contains a
comma too much.

--Alex

On Thu, May 1, 2014 at 1:53 AM, Deepak Jha dkjhanitt@gmail.com wrote:

Hi,
I have setup ELK stack and I am going by default index name, which is
logstash-YYYY.MM.DD . Since this is the only index format I have, I decided
to create a template file, so that whenever new index gets created i can
set up the mapping property. I am not able to push the data to
elasticsearch if my index mapping gets created from template. May I know
where am I wrong ?

Here is my mapping file content:
{
"X_Server" : {
"properties" : {
"@timestamp" : {
"type" : "date",
"format" : "dateOptionalTime"
},
"@version" : { "type" : "string" },
"class" : { "type" : "string" },
"file" : { "type" : "string"},
"message": {"type": "string"},
"host" : { "type" : "string", "index": "not_analyzed" }
}}}

My template file content is

{
"template": "logstash-*",
"settings" : {
"index.number_of_shards" : 3,
"index.number_of_replicas" : 1,
"index.query.default_field" : "@message",
"index.routing.allocation.total_shards_per_node" : 2,
"index.auto_expand_replicas": false
},
"mappings": {
"X_Server": {
"_all": { "enabled": false },
"_source": { "compress": false },
"properties" : {
"class" : { "type" : "string", },
"host" : { "type" : "string", "index" : "not_analyzed" },
"file" : { "type" : "string" },
"message" : { "type": "string"}
}
}}}

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/b1d382b5-0fa7-4a2c-96f0-150d856482cc%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/b1d382b5-0fa7-4a2c-96f0-150d856482cc%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGCwEM8%2BeqGWt9H%2BbJKLg0B5KfmOSjyGhfDXvWFdQgaR4tWQ4w%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

3

Hi Alex,
I am using logstash to index my data to elasticsearch. When I do not use
this template I am able to push data.
Here is the few lines from the log file that I have:

2014-05-05 14:51:20,248 (main) a_class INFO: message1
2014-05-05 14:51:20,249 (main) a_class INFO: message2
2014-05-05 14:51:20,510 (main) b_class INFO: message3
2014-05-05 14:51:20,597 (main) x_class INFO: message4
2014-05-05 14:51:20,701 (main) d_class INFO: message5

My grok parser is
SERVERLOG %{TIMESTAMP_ISO8601} (%{NOTSPACE:thread}) %{WORD:class}
%{LOGLEVEL:severity}: %{GREEDYDATA:message}

Attached my template.json.. Thanks a lot for your help.

I am processing these in logstash
On Monday, May 5, 2014 3:37:52 AM UTC-7, Alexander Reelsen wrote:

Hey,

can you include a full example including data you have been indexing? Or
include error messages, as you wrote you cannot index any data anymore?

Also, your JSON is not valid as the mapping for the class field contains a
comma too much.

--Alex

On Thu, May 1, 2014 at 1:53 AM, Deepak Jha <dkjh...@gmail.com<javascript:>

wrote:

Hi,
I have setup ELK stack and I am going by default index name, which is
logstash-YYYY.MM.DD . Since this is the only index format I have, I decided
to create a template file, so that whenever new index gets created i can
set up the mapping property. I am not able to push the data to
elasticsearch if my index mapping gets created from template. May I know
where am I wrong ?

Here is my mapping file content:
{
"X_Server" : {
"properties" : {
"@timestamp" : {
"type" : "date",
"format" : "dateOptionalTime"
},
"@version" : { "type" : "string" },
"class" : { "type" : "string" },
"file" : { "type" : "string"},
"message": {"type": "string"},
"host" : { "type" : "string", "index": "not_analyzed" }
}}}

My template file content is

{
"template": "logstash-*",
"settings" : {
"index.number_of_shards" : 3,
"index.number_of_replicas" : 1,
"index.query.default_field" : "@message",
"index.routing.allocation.total_shards_per_node" : 2,
"index.auto_expand_replicas": false
},
"mappings": {
"X_Server": {
"_all": { "enabled": false },
"_source": { "compress": false },
"properties" : {
"class" : { "type" : "string", },
"host" : { "type" : "string", "index" : "not_analyzed" },
"file" : { "type" : "string" },
"message" : { "type": "string"}
}
}}}

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/b1d382b5-0fa7-4a2c-96f0-150d856482cc%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/b1d382b5-0fa7-4a2c-96f0-150d856482cc%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/2b5c8869-743e-4401-829e-6cac4ccbab75%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

4

Hey,

can you just take some sample data and index it into elasticsearch manually
and see if that works?

--Alex

On Thu, May 1, 2014 at 1:53 AM, Deepak Jha dkjhanitt@gmail.com wrote:

Hi,
I have setup ELK stack and I am going by default index name, which is
logstash-YYYY.MM.DD . Since this is the only index format I have, I decided
to create a template file, so that whenever new index gets created i can
set up the mapping property. I am not able to push the data to
elasticsearch if my index mapping gets created from template. May I know
where am I wrong ?

Here is my mapping file content:
{
"X_Server" : {
"properties" : {
"@timestamp" : {
"type" : "date",
"format" : "dateOptionalTime"
},
"@version" : { "type" : "string" },
"class" : { "type" : "string" },
"file" : { "type" : "string"},
"message": {"type": "string"},
"host" : { "type" : "string", "index": "not_analyzed" }
}}}

My template file content is

{
"template": "logstash-*",
"settings" : {
"index.number_of_shards" : 3,
"index.number_of_replicas" : 1,
"index.query.default_field" : "@message",
"index.routing.allocation.total_shards_per_node" : 2,
"index.auto_expand_replicas": false
},
"mappings": {
"X_Server": {
"_all": { "enabled": false },
"_source": { "compress": false },
"properties" : {
"class" : { "type" : "string", },
"host" : { "type" : "string", "index" : "not_analyzed" },
"file" : { "type" : "string" },
"message" : { "type": "string"}
}
}}}

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/b1d382b5-0fa7-4a2c-96f0-150d856482cc%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/b1d382b5-0fa7-4a2c-96f0-150d856482cc%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAGCwEM8YwuhQBz6Th%2B0VsmUrRwZXnVqHe8SwV_2gemyQ3oURJQ%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

5

Hi Alexander,
Yes it works when I remove the template setting.

On Friday, May 9, 2014 12:26:49 PM UTC-7, Alexander Reelsen wrote:

Hey,

can you just take some sample data and index it into elasticsearch
manually and see if that works?

--Alex

On Thu, May 1, 2014 at 1:53 AM, Deepak Jha <dkjh...@gmail.com<javascript:>

wrote:

Hi,
I have setup ELK stack and I am going by default index name, which is
logstash-YYYY.MM.DD . Since this is the only index format I have, I decided
to create a template file, so that whenever new index gets created i can
set up the mapping property. I am not able to push the data to
elasticsearch if my index mapping gets created from template. May I know
where am I wrong ?

Here is my mapping file content:
{
"X_Server" : {
"properties" : {
"@timestamp" : {
"type" : "date",
"format" : "dateOptionalTime"
},
"@version" : { "type" : "string" },
"class" : { "type" : "string" },
"file" : { "type" : "string"},
"message": {"type": "string"},
"host" : { "type" : "string", "index": "not_analyzed" }
}}}

My template file content is

{
"template": "logstash-*",
"settings" : {
"index.number_of_shards" : 3,
"index.number_of_replicas" : 1,
"index.query.default_field" : "@message",
"index.routing.allocation.total_shards_per_node" : 2,
"index.auto_expand_replicas": false
},
"mappings": {
"X_Server": {
"_all": { "enabled": false },
"_source": { "compress": false },
"properties" : {
"class" : { "type" : "string", },
"host" : { "type" : "string", "index" : "not_analyzed" },
"file" : { "type" : "string" },
"message" : { "type": "string"}
}
}}}

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/b1d382b5-0fa7-4a2c-96f0-150d856482cc%40googlegroups.comhttps://groups.google.com/d/msgid/elasticsearch/b1d382b5-0fa7-4a2c-96f0-150d856482cc%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/76e896a0-8e74-417b-8027-63b3fe67f2bc%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.