How to create indexes for each tenants?

Elastic Stack Elasticsearch
#1

Hi Guys,

Right now i have 1 index with 5 shards which is storing the multiple tenant logs. now I want to create indexes in elastic cluster for each tenant and I need to keep the docs for 1 year or 6 month time period based on requirement. splunk forwarder is my default log forwarder for splunk as well as graylog+elastic cluster. please help me to create individual indexes for tenants and how to redirect the log to respective tenant index.

#2

Please format you post correctly. As it is now it is very difficult to read. If you want to index different tenants into separate indices you usually need to address that in your ingestion pipeline, so it looks like this post might be filed under the wrong category.

How many tenants do you have? How much data are you ingesting per day?

#3

I have 14 tenants and nearly 6 to 8 GB of data is ingesting per day and number of tenants may increase up to 300 in future.

#4

Each shard in Elasticsearch carries with it a certain amount of overhead in terms of memory usage and file handles. Having a large number of very small indices is therefore quite inefficient use of resources. Given the small daily data volumes I would recommend considering keeping all users in a shared index, possibly together with the use of routing.

If you still need to separate out the users into their own indices, I would recommend setting up monthly indices, each backed by a single shard. Going with daily indices for that number of users and retention period would most likely result in too many small shards.

When it comes to how to do this in your ingest pipeline I will not be able to help as I have no experience with Spunk nor Graylog.

#5