Query and allocate data to shards based on tags

Elastic Stack Elasticsearch
1

Hi,

I'm fairly new to elasticsearch, so sorry if this is a trivial question.

I'm running a typical logstash-redis-elasticsearch system to capture all my logs(around 500 GB/day). To my knowledge elasticsearch queries every shard in an index and aggregates the results, but due to the volume of logs per day and the response times needed, I want to query only few shards which of course should be decided on some "tag" in the message. So I'm looking at a way to allocate data to shards based on some tags and query only relevant shards based on the tags. Any leads, references or solutions on how to achieve this ?

I've already looked at shard allocation filtering but that doesn't cater this specific requirement.

Thanks,
Aryan

2

What you want is called "routing" in elasticsearch. Search for it by that
name and you will find plenty of information.

On Tue, Jul 2, 2013 at 11:01 PM, aryan abhilashm24@yahoo.com wrote:

Hi,

I'm fairly new to elasticsearch, so sorry if this is a trivial question.

I'm running a typical logstash-redis-elasticsearch system to capture all my
logs(around 500 GB/day). To my knowledge elasticsearch queries every shard
in an index and aggregates the results, but due to the volume of logs per
day and the response times needed, I want to query only few shards which of
course should be decided on some "tag" in the message. So I'm looking at a
way to allocate data to shards based on some tags and query only relevant
shards based on the tags. Any leads, references or solutions on how to
achieve this ?

I've already looked at shard allocation filtering but that doesn't cater
this specific requirement.

Thanks,
Aryan

--
View this message in context:
http://elasticsearch-users.115913.n3.nabble.com/Query-and-allocate-data-to-shards-based-on-tags-tp4037420.html
Sent from the Elasticsearch Users mailing list archive at Nabble.com.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

3

Hey,

another option are time based indices (especially in a log file based
setup), as you are searching for a certain time period most of the time.

--Alex

On Wed, Jul 10, 2013 at 1:10 AM, Randall McRee randall.mcree@gmail.comwrote:

What you want is called "routing" in elasticsearch. Search for it by that
name and you will find plenty of information.

On Tue, Jul 2, 2013 at 11:01 PM, aryan abhilashm24@yahoo.com wrote:

Hi,

I'm fairly new to elasticsearch, so sorry if this is a trivial question.

I'm running a typical logstash-redis-elasticsearch system to capture all
my
logs(around 500 GB/day). To my knowledge elasticsearch queries every shard
in an index and aggregates the results, but due to the volume of logs per
day and the response times needed, I want to query only few shards which
of
course should be decided on some "tag" in the message. So I'm looking at a
way to allocate data to shards based on some tags and query only relevant
shards based on the tags. Any leads, references or solutions on how to
achieve this ?

I've already looked at shard allocation filtering but that doesn't cater
this specific requirement.

Thanks,
Aryan

--
View this message in context:
http://elasticsearch-users.115913.n3.nabble.com/Query-and-allocate-data-to-shards-based-on-tags-tp4037420.html
Sent from the Elasticsearch Users mailing list archive at Nabble.com.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.