[SOLVED] Customing document routing

Elastic Stack Elasticsearch
1

Hi all,

Regarding elasticsearch blog I can read:

"Routing ensures that all documents with the same routing value will locate to the same shard"

Now in my configuration I have 20 shards and 4 document type. It is possible to route a document type toward 5 shards, a second document type toward another 5 shards, etc ?

Because I don't want to route a document type toward just one shard and get others document in the 19 shards left or inside the first shard, according the blog:

"Routing ensures that documents of a particular routing value all go to the same shard…but that doesn’t mean that other documents aren’t routed to the shard too."

Thanks in advance,
Alex

2

No. That's not possible IMO.
The routing key is used to compute a single shard number.

You could use routing keys like foo1, foo2, foo3 for all docs of foo type but this is something you have to deal with on a client level.

Why do you have 20 shards BTW?

3

Hi,

Thanks for your answer.

I have 20 shards because I have 6 nodes in my cluster, it will be possible to increase the node number to 10.
So the goal is to have 2 shards per node (2 processes running on each node).

Have a good day,
Alex

4

what kind of data are you indexing?
Depending on that you could imagine another way to design your indices.

5

It's logs.

I'm interesting :slight_smile:

6

Logs commonly have timestamp. A good practice is to create rolling indices. One per timeframe you wish:
day, month, year, 10d... Whatever.

It means that you can easily drop old data with just a DELETE oldindex call. You can create an alias on top of them so you will query all the indices as if there was only one.

Well. Then if you feel that you need more horse power, just start new nodes and let elasticsearch do its job.

Have a look at https://www.elastic.co/guide/en/elasticsearch/guide/current/time-based.html

7

Hi thanks,

That's already what I did :wink:

I have 3 nodes levels:

  • hot
  • cold
  • archive

I route indices between hot and cold nodes and between cold and archive nodes I make a reindex with a lost data and segment optimization.

Have a good day,
Alex