Creating index dynamically in ES

Elastic Stack Elasticsearch
1

Wondering if i could create index based on the JSON fields dynamically in
Elasticsearch when Logstash forwards a JSON formatted message to
Elasticsearch.

For example:

Logstash forwards message in the following format:

"MONTH": [
[
"Jan"
]
],
"MONTHDAY": [
[
"9"
]
]

I wanted to create index based on MONTH and MONTHDAY in Elasticsearch. Could someone please help me out?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/bce88ea2-1ec4-49ff-9101-af63e36fec85%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

2

You can specify the index name in the elasticsearch output:

http://logstash.net/docs/1.3.3/outputs/elasticsearch#index

For example, let's say I have a field named clientip, I can make indexes
named ls-clientip by specifying something like this:

output {
elasticsearch {
host => "localhost"
index => "ls-%{clientip}"
}
}

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/8ed560c0-1dde-45d3-a866-85491670fdd1%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

3

Thanks for the response.

I was not looking for this one.

What i meant was could i create indexes based on MONTH and MONTHDAY
respectively at the same time.

On Tuesday, February 18, 2014 11:01:59 AM UTC+5:30, san wrote:

Wondering if i could create index based on the JSON fields dynamically in
Elasticsearch when Logstash forwards a JSON formatted message to
Elasticsearch.

For example:

Logstash forwards message in the following format:

"MONTH": [
[
"Jan"
]
],
"MONTHDAY": [
[
"9"
]
]

I wanted to create index based on MONTH and MONTHDAY in Elasticsearch. Could someone please help me out?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/09dbcfaa-5296-4170-9809-432f372060b7%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

4

Keep in mind that ES index names can only be all lowercase (so you have to
lowercase the values first in LS). Try something like this (after
lowercasing the values first)?

output {
elasticsearch {
host => "localhost"
index => "%{MONTH}-%{MONTHDAY}"
}
}

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/bed282eb-3ad9-41f6-b692-0e5acbe295ad%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

5

Hello Binh Ly,
How or where do you declare the variables such as %{clientip}?
Thanks

El miércoles, 19 de febrero de 2014 05:02:40 UTC+13, Binh Ly escribió:

You can specify the index name in the elasticsearch output:

Elasticsearch output plugin | Logstash Reference [8.11] | Elastic

For example, let's say I have a field named clientip, I can make indexes
named ls-clientip by specifying something like this:

output {
elasticsearch {
host => "localhost"
index => "ls-%{clientip}"
}
}

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/9d47dbd1-fcbd-4ade-8f9f-ecfc5a622415%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

6

On Tuesday, November 04, 2014 at 00:57 CET,
Alejandro Alves alejandro.alves@gmail.com wrote:

El miércoles, 19 de febrero de 2014 05:02:40 UTC+13, Binh Ly
escribió:

You can specify the index name in the elasticsearch output:
Elasticsearch output plugin | Logstash Reference [8.11] | Elastic
For example, let's say I have a field named clientip, I can make
indexes named ls-clientip by specifying something like this:
output {
elasticsearch {
host => "localhost"
index => "ls-%{clientip}"
}
}

How or where do you declare the variables such as %{clientip}?

They are fields in the message, often populated by a 'grok' filter.

Note that not all strings in Logstash configuration files support
%{variablename} interpolation. Where you can use such references
is unfortunately underdocumented.

--
Magnus Bäck | Software Engineer, Development Tools
magnus.back@sonymobile.com | Sony Mobile Communications

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/20141110075747.GD6370%40seldlx20533.corpusers.net.
For more options, visit https://groups.google.com/d/optout.