Query and allocate data to shards based on tags

Hi,

I'm fairly new to elasticsearch, so sorry if this is a trivial question.

I'm running a typical logstash-redis-elasticsearch system to capture all my logs(around 500 GB/day). To my knowledge elasticsearch queries every shard in an index and aggregates the results, but due to the volume of logs per day and the response times needed, I want to query only few shards which of course should be decided on some "tag" in the message. So I'm looking at a way to allocate data to shards based on some tags and query only relevant shards based on the tags. Any leads, references or solutions on how to achieve this ?

I've already looked at shard allocation filtering but that doesn't cater this specific requirement.

Thanks,
Aryan

What you want is called "routing" in elasticsearch. Search for it by that
name and you will find plenty of information.

On Tue, Jul 2, 2013 at 11:01 PM, aryan abhilashm24@yahoo.com wrote:

Hi,

I'm fairly new to elasticsearch, so sorry if this is a trivial question.

I'm running a typical logstash-redis-elasticsearch system to capture all my
logs(around 500 GB/day). To my knowledge elasticsearch queries every shard
in an index and aggregates the results, but due to the volume of logs per
day and the response times needed, I want to query only few shards which of
course should be decided on some "tag" in the message. So I'm looking at a
way to allocate data to shards based on some tags and query only relevant
shards based on the tags. Any leads, references or solutions on how to
achieve this ?

I've already looked at shard allocation filtering but that doesn't cater
this specific requirement.

Thanks,
Aryan

--
View this message in context:
http://elasticsearch-users.115913.n3.nabble.com/Query-and-allocate-data-to-shards-based-on-tags-tp4037420.html
Sent from the ElasticSearch Users mailing list archive at Nabble.com.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

Hey,

another option are time based indices (especially in a log file based
setup), as you are searching for a certain time period most of the time.

--Alex

On Wed, Jul 10, 2013 at 1:10 AM, Randall McRee randall.mcree@gmail.comwrote:

What you want is called "routing" in elasticsearch. Search for it by that
name and you will find plenty of information.

On Tue, Jul 2, 2013 at 11:01 PM, aryan abhilashm24@yahoo.com wrote:

Hi,

I'm fairly new to elasticsearch, so sorry if this is a trivial question.

I'm running a typical logstash-redis-elasticsearch system to capture all
my
logs(around 500 GB/day). To my knowledge elasticsearch queries every shard
in an index and aggregates the results, but due to the volume of logs per
day and the response times needed, I want to query only few shards which
of
course should be decided on some "tag" in the message. So I'm looking at a
way to allocate data to shards based on some tags and query only relevant
shards based on the tags. Any leads, references or solutions on how to
achieve this ?

I've already looked at shard allocation filtering but that doesn't cater
this specific requirement.

Thanks,
Aryan

--
View this message in context:
http://elasticsearch-users.115913.n3.nabble.com/Query-and-allocate-data-to-shards-based-on-tags-tp4037420.html
Sent from the ElasticSearch Users mailing list archive at Nabble.com.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups
"elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an
email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.

--
You received this message because you are subscribed to the Google Groups "elasticsearch" group.
To unsubscribe from this group and stop receiving emails from it, send an email to elasticsearch+unsubscribe@googlegroups.com.
For more options, visit https://groups.google.com/groups/opt_out.