Elasticsearch index name question

I use a Monitoring Framework designed as a solution to monitor
heterogeneous networks and systems in terms of services (platforms,
applications for TELCO systems).

This framework collects in synchronous way the required data from several
devices, it stores them in a Mongo Data Base and then transfers all stored
collections from MongoDB to Elasticsearch via river-mongodb plugin.

We can have a huge amount of data stored in a single index of
Elasticsearch, for example, about 5.2 millions of documents can be
collected in a single MongoDB collection for only 8 hours of monitoring and
so the number of documents in a single index grows rapidly.

At present, I have installed on Centos 6.5 server an Elasticsearch Cluster
configuration with one node and five indices but only one index for all
synchronous data.

My problem is to be able to create different indices in Elasticsearch where
I can share the synchronous data, and so I would like to know if it is
possible to create an index name with a timestamp appended to it, like so
Logstash uses the timestamp from an event to derive the related
Elasticsearch index name.

Some idea, suggestion, help?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/37bb3713-56d9-443c-b3a5-9056092b958d%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

This is possible but not automatically within ES.
LS knows it needs to switch to a new index at 0000UTC, you need to find a
way to get the river or some other code to do this.

On 19 February 2015 at 21:59, Silvana Vezzoli silvana.vezzoli@gmail.com
wrote:

I use a Monitoring Framework designed as a solution to monitor
heterogeneous networks and systems in terms of services (platforms,
applications for TELCO systems).

This framework collects in synchronous way the required data from several
devices, it stores them in a Mongo Data Base and then transfers all stored
collections from MongoDB to Elasticsearch via river-mongodb plugin.

We can have a huge amount of data stored in a single index of
Elasticsearch, for example, about 5.2 millions of documents can be
collected in a single MongoDB collection for only 8 hours of monitoring and
so the number of documents in a single index grows rapidly.

At present, I have installed on Centos 6.5 server an Elasticsearch Cluster
configuration with one node and five indices but only one index for all
synchronous data.

My problem is to be able to create different indices in Elasticsearch
where I can share the synchronous data, and so I would like to know if it
is possible to create an index name with a timestamp appended to it, like
so Logstash uses the timestamp from an event to derive the related
Elasticsearch index name.

Some idea, suggestion, help?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/37bb3713-56d9-443c-b3a5-9056092b958d%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/37bb3713-56d9-443c-b3a5-9056092b958d%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X-cqwvKHdQETdZtWOTOUmwPWayYWKiNwmo6JXETno0%3DEg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.

Thanks, Mark.
But supposing that I find the way to get the river to filter the documents
to be indexed in ES on a daily base (for example), my question is if it is
possible to create a dynamic index name in Elasticsearch based on some
variable value (curl -XPUT 'http://localhost:9200/
${elasticsearch.index.name}/').
It seems that at present itsn't possible, do it could be a new
Elasticsearch feature?

Il giorno giovedì 19 febbraio 2015 23:02:12 UTC+1, Mark Walkom ha scritto:

This is possible but not automatically within ES.
LS knows it needs to switch to a new index at 0000UTC, you need to find a
way to get the river or some other code to do this.

On 19 February 2015 at 21:59, Silvana Vezzoli <silvana...@gmail.com
<javascript:>> wrote:

I use a Monitoring Framework designed as a solution to monitor
heterogeneous networks and systems in terms of services (platforms,
applications for TELCO systems).

This framework collects in synchronous way the required data from
several devices, it stores them in a Mongo Data Base and then transfers all
stored collections from MongoDB to Elasticsearch via river-mongodb plugin.

We can have a huge amount of data stored in a single index of
Elasticsearch, for example, about 5.2 millions of documents can be
collected in a single MongoDB collection for only 8 hours of monitoring and
so the number of documents in a single index grows rapidly.

At present, I have installed on Centos 6.5 server an Elasticsearch
Cluster configuration with one node and five indices but only one index for
all synchronous data.

My problem is to be able to create different indices in Elasticsearch
where I can share the synchronous data, and so I would like to know if it
is possible to create an index name with a timestamp appended to it, like
so Logstash uses the timestamp from an event to derive the related
Elasticsearch index name.

Some idea, suggestion, help?

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearc...@googlegroups.com <javascript:>.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/37bb3713-56d9-443c-b3a5-9056092b958d%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/37bb3713-56d9-443c-b3a5-9056092b958d%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/eca8a3b0-dbf4-46f6-894f-1b73458335be%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.

You could use index templates -
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/indices-templates.html

On 24 February 2015 at 00:35, svezzoli silvana.vezzoli@gmail.com wrote:

Thanks, Mark.
But supposing that I find the way to get the river to filter the documents
to be indexed in ES on a daily base (for example), my question is if it is
possible to create a dynamic index name in Elasticsearch based on some
variable value (curl -XPUT 'http://localhost:9200/${
elasticsearch.index.name}/').
It seems that at present itsn't possible, do it could be a new
Elasticsearch feature?

Il giorno giovedì 19 febbraio 2015 23:02:12 UTC+1, Mark Walkom ha scritto:

This is possible but not automatically within ES.
LS knows it needs to switch to a new index at 0000UTC, you need to find a
way to get the river or some other code to do this.

On 19 February 2015 at 21:59, Silvana Vezzoli silvana...@gmail.com
wrote:

I use a Monitoring Framework designed as a solution to monitor
heterogeneous networks and systems in terms of services (platforms,
applications for TELCO systems).

This framework collects in synchronous way the required data from
several devices, it stores them in a Mongo Data Base and then transfers all
stored collections from MongoDB to Elasticsearch via river-mongodb plugin.

We can have a huge amount of data stored in a single index of
Elasticsearch, for example, about 5.2 millions of documents can be
collected in a single MongoDB collection for only 8 hours of monitoring and
so the number of documents in a single index grows rapidly.

At present, I have installed on Centos 6.5 server an Elasticsearch
Cluster configuration with one node and five indices but only one index for
all synchronous data.

My problem is to be able to create different indices in Elasticsearch
where I can share the synchronous data, and so I would like to know if it
is possible to create an index name with a timestamp appended to it, like
so Logstash uses the timestamp from an event to derive the related
Elasticsearch index name.

Some idea, suggestion, help?

--
You received this message because you are subscribed to the Google
Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send
an email to elasticsearc...@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/
msgid/elasticsearch/37bb3713-56d9-443c-b3a5-9056092b958d%
40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/37bb3713-56d9-443c-b3a5-9056092b958d%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit
https://groups.google.com/d/msgid/elasticsearch/eca8a3b0-dbf4-46f6-894f-1b73458335be%40googlegroups.com
https://groups.google.com/d/msgid/elasticsearch/eca8a3b0-dbf4-46f6-894f-1b73458335be%40googlegroups.com?utm_medium=email&utm_source=footer
.
For more options, visit https://groups.google.com/d/optout.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/CAEYi1X8YaFvF%2Bbz-LE8%3D1q%3DpTCU1PnaDmUUgxn4B2fBf-MaKzg%40mail.gmail.com.
For more options, visit https://groups.google.com/d/optout.