Unique index for each user

Elastic Stack Elasticsearch
1

Hi everyone.
I'm sending my application's logs to logstash as json and I want to save them in elasticsearch by different indices.
In fact, I want to save logs of each user in a unique index that is equal to user key in the incoming json document. Because I want to provide different services to different users.
For example:

  • document storage time and also services for users can vary.
  • log content of each user may increase and I intend to increase the query speed by separating the indices.

Therefore:
Q1: Is it a good idea to separate each user's index?
Q2: How can I do it in logstash output config file?

2

Having lots of small indices and shards is very inefficient and does not d ale well. Separating indices per user can work for relatively small number of users though. If you however have lots of small users it is generally better to have them share indices and filter the queries.

How many users do you expect to need to support?

3

About 100,000 users. So do you recommend have same index for all users?

4

With that amount of users you need to at least group them and could possibly benefit from using routing.

5

@Christian_Dahlqvist
Thank you.

6

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.