I use a Monitoring Framework designed as a solution to monitor
heterogeneous networks and systems in terms of services (platforms,
applications for TELCO systems).
This framework collects in synchronous way the required data from several
devices, it stores them in a Mongo Data Base and then transfers all stored
collections from MongoDB to Elasticsearch via river-mongodb plugin.
We can have a huge amount of data stored in a single index of
Elasticsearch, for example, about 5.2 millions of documents can be
collected in a single MongoDB collection for only 8 hours of monitoring and
so the number of documents in a single index grows rapidly.
At present, I have installed on Centos 6.5 server an Elasticsearch Cluster
configuration with one node and five indices but only one index for all
synchronous data.
My problem is to be able to create different indices in Elasticsearch where
I can share the synchronous data, and so I would like to know if it is
possible to create an index name with a timestamp appended to it, like so
Logstash uses the timestamp from an event to derive the related
Elasticsearch index name.
This is possible but not automatically within ES.
LS knows it needs to switch to a new index at 0000UTC, you need to find a
way to get the river or some other code to do this.
I use a Monitoring Framework designed as a solution to monitor
heterogeneous networks and systems in terms of services (platforms,
applications for TELCO systems).
This framework collects in synchronous way the required data from several
devices, it stores them in a Mongo Data Base and then transfers all stored
collections from MongoDB to Elasticsearch via river-mongodb plugin.
We can have a huge amount of data stored in a single index of
Elasticsearch, for example, about 5.2 millions of documents can be
collected in a single MongoDB collection for only 8 hours of monitoring and
so the number of documents in a single index grows rapidly.
At present, I have installed on Centos 6.5 server an Elasticsearch Cluster
configuration with one node and five indices but only one index for all
synchronous data.
My problem is to be able to create different indices in Elasticsearch
where I can share the synchronous data, and so I would like to know if it
is possible to create an index name with a timestamp appended to it, like
so Logstash uses the timestamp from an event to derive the related
Elasticsearch index name.
Thanks, Mark.
But supposing that I find the way to get the river to filter the documents
to be indexed in ES on a daily base (for example), my question is if it is
possible to create a dynamic index name in Elasticsearch based on some
variable value (curl -XPUT 'http://localhost:9200/
${elasticsearch.index.name}/').
It seems that at present itsn't possible, do it could be a new
Elasticsearch feature?
Il giorno giovedì 19 febbraio 2015 23:02:12 UTC+1, Mark Walkom ha scritto:
This is possible but not automatically within ES.
LS knows it needs to switch to a new index at 0000UTC, you need to find a
way to get the river or some other code to do this.
On 19 February 2015 at 21:59, Silvana Vezzoli <silvana...@gmail.com
<javascript:>> wrote:
I use a Monitoring Framework designed as a solution to monitor
heterogeneous networks and systems in terms of services (platforms,
applications for TELCO systems).
This framework collects in synchronous way the required data from
several devices, it stores them in a Mongo Data Base and then transfers all
stored collections from MongoDB to Elasticsearch via river-mongodb plugin.
We can have a huge amount of data stored in a single index of
Elasticsearch, for example, about 5.2 millions of documents can be
collected in a single MongoDB collection for only 8 hours of monitoring and
so the number of documents in a single index grows rapidly.
At present, I have installed on Centos 6.5 server an Elasticsearch
Cluster configuration with one node and five indices but only one index for
all synchronous data.
My problem is to be able to create different indices in Elasticsearch
where I can share the synchronous data, and so I would like to know if it
is possible to create an index name with a timestamp appended to it, like
so Logstash uses the timestamp from an event to derive the related
Elasticsearch index name.
Thanks, Mark.
But supposing that I find the way to get the river to filter the documents
to be indexed in ES on a daily base (for example), my question is if it is
possible to create a dynamic index name in Elasticsearch based on some
variable value (curl -XPUT 'http://localhost:9200/${
elasticsearch.index.name}/').
It seems that at present itsn't possible, do it could be a new
Elasticsearch feature?
Il giorno giovedì 19 febbraio 2015 23:02:12 UTC+1, Mark Walkom ha scritto:
This is possible but not automatically within ES.
LS knows it needs to switch to a new index at 0000UTC, you need to find a
way to get the river or some other code to do this.
I use a Monitoring Framework designed as a solution to monitor
heterogeneous networks and systems in terms of services (platforms,
applications for TELCO systems).
This framework collects in synchronous way the required data from
several devices, it stores them in a Mongo Data Base and then transfers all
stored collections from MongoDB to Elasticsearch via river-mongodb plugin.
We can have a huge amount of data stored in a single index of
Elasticsearch, for example, about 5.2 millions of documents can be
collected in a single MongoDB collection for only 8 hours of monitoring and
so the number of documents in a single index grows rapidly.
At present, I have installed on Centos 6.5 server an Elasticsearch
Cluster configuration with one node and five indices but only one index for
all synchronous data.
My problem is to be able to create different indices in Elasticsearch
where I can share the synchronous data, and so I would like to know if it
is possible to create an index name with a timestamp appended to it, like
so Logstash uses the timestamp from an event to derive the related
Elasticsearch index name.
Apache, Apache Lucene, Apache Hadoop, Hadoop, HDFS and the yellow elephant
logo are trademarks of the
Apache Software Foundation
in the United States and/or other countries.