Multiple indices in elastic search with logstash output


(f6) #1

I am using logstash+ elastic-search+kibana framework for my project. I want
to create per tenant index in Elastic search. Is this a efficient way to
do?

Can i customize elastic search to look for a specific tag in logstash
output and create and store the data in particular index?

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Radu Gheorghe) #2

Hello,

On Sun, May 12, 2013 at 8:01 AM, mailtov16@gmail.com wrote:

I am using logstash+ elastic-search+kibana framework for my project. I
want to create per tenant index in Elastic search. Is this a efficient way
to do?

I think it depends on how many indices you'll end up having. If you'll have
a huge amount of indices, they'll use up more space, more memory and more
file descriptors. Because you'll have more shards, and therefore more
segments. Although, you can tune the number of shards and the merge
policyhttp://www.elasticsearch.org/guide/reference/index-modules/merge/to
have less segments.

However, when you search for data from a single tenant, it should be faster
than if you mix all data up and do the same search by filtering on a field.
Because there's less data to search on.

Can i customize elastic search to look for a specific tag in logstash
output and create and store the data in particular index?

I'm not sure exactly what you're after, but it sounds like you want to
index different types of data in different places. If that's so, I think
logstash is the place where you can configure, for example, the destination
index name to depend on a variable. People on the Logstash mailing list
will definitely know more about what you can do here.

Best regards,
Radu

http://sematext.com/ -- ElasticSearch -- Solr -- Lucene

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(f6) #3

Thnx Radu

On Mon, May 13, 2013 at 12:16 PM, Radu Gheorghe
radu.gheorghe@sematext.comwrote:

Hello,

On Sun, May 12, 2013 at 8:01 AM, mailtov16@gmail.com wrote:

I am using logstash+ elastic-search+kibana framework for my project. I
want to create per tenant index in Elastic search. Is this a efficient way
to do?

I think it depends on how many indices you'll end up having. If you'll
have a huge amount of indices, they'll use up more space, more memory and
more file descriptors. Because you'll have more shards, and therefore more
segments. Although, you can tune the number of shards and the merge policyhttp://www.elasticsearch.org/guide/reference/index-modules/merge/to have less segments.

However, when you search for data from a single tenant, it should be
faster than if you mix all data up and do the same search by filtering on a
field. Because there's less data to search on.

Can i customize elastic search to look for a specific tag in logstash
output and create and store the data in particular index?

I'm not sure exactly what you're after, but it sounds like you want to
index different types of data in different places. If that's so, I think
logstash is the place where you can configure, for example, the destination
index name to depend on a variable. People on the Logstash mailing list
will definitely know more about what you can do here.

Best regards,
Radu

http://sematext.com/ -- ElasticSearch -- Solr -- Lucene

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/zss4kvLMkNw/unsubscribe?hl=en-US
.
To unsubscribe from this group and all its topics, send an email to
elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(Jaap H) #4

I was just wondering if you have succeeded in creating the multi tenant
indices?
What I'd want to avoid is to run as many agents/indexers as I have
tenants...

Op dinsdag 14 mei 2013 06:31:22 UTC+2 schreef freak 62:

Thnx Radu

On Mon, May 13, 2013 at 12:16 PM, Radu Gheorghe <radu.g...@sematext.com<javascript:>

wrote:

Hello,

On Sun, May 12, 2013 at 8:01 AM, <mail...@gmail.com <javascript:>> wrote:

I am using logstash+ elastic-search+kibana framework for my project. I
want to create per tenant index in Elastic search. Is this a efficient way
to do?

I think it depends on how many indices you'll end up having. If you'll
have a huge amount of indices, they'll use up more space, more memory and
more file descriptors. Because you'll have more shards, and therefore more
segments. Although, you can tune the number of shards and the merge
policyhttp://www.elasticsearch.org/guide/reference/index-modules/merge/to have less segments.

However, when you search for data from a single tenant, it should be
faster than if you mix all data up and do the same search by filtering on a
field. Because there's less data to search on.

Can i customize elastic search to look for a specific tag in logstash
output and create and store the data in particular index?

I'm not sure exactly what you're after, but it sounds like you want to
index different types of data in different places. If that's so, I think
logstash is the place where you can configure, for example, the destination
index name to depend on a variable. People on the Logstash mailing list
will definitely know more about what you can do here.

Best regards,
Radu

http://sematext.com/ -- ElasticSearch -- Solr -- Lucene

--
You received this message because you are subscribed to a topic in the
Google Groups "elasticsearch" group.
To unsubscribe from this topic, visit
https://groups.google.com/d/topic/elasticsearch/zss4kvLMkNw/unsubscribe?hl=en-US
.
To unsubscribe from this group and all its topics, send an email to
elasticsearc...@googlegroups.com <javascript:>.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
To view this discussion on the web visit https://groups.google.com/d/msgid/elasticsearch/2ecfe37f-141a-4b3c-80e0-fb6176ff5a66%40googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.


(system) #5