Currently we are using ElasticSearch v2.0 and we are making use of this to read our application log files and create indexes using ElasticSearch. We have read of using Routing to enhance performance.
We were looking at ways to route data using a field other than uid. Any ways to route based on a field in the log file using ElasticSearch v2.0, so that fields of a particular type are routed to a particular shard.
Hi,
You need to pass the value at index & query time as a parameter. In Elasticsearch 2.0 you can't use a field to extract the routing id. However you may want to re-think using document routing since ultimately what can happen is that the distribution isn't even. So some shards will be much larger then others and you end up with hot spots where nodes with larger shards end up doing most of the work. Then you need to re-index your data to put larger users in their own index.
Like Mike alluded to, I wouldn't recommend routing for this use case.
Thanks for the response. Just to add a bit more of details. I am using FluentD to collect data and send it across to elasticSearch using the plugin https://github.com/uken/fluent-plugin-elasticsearch. In that case is there a way to configure the document routing.
The fluentD configuration is as follows:
type elasticsearch
include_tag_key true
tag_key tag
logstash_format true
logstash_prefix logtest4
hosts a.b.c.d:9200,a.b.c.d:9200
#buffering
buffer_type file
buffer_path /datadrive/buffer/td
buffer_chunk_limit 16m
buffer_queue_limit 16384
flush_interval 5m
retry_wait 15s
request_timeout 15s
num_threads 2
type_name log
log_level trace
That doesn't really change the recommendations 