Single index or multiple index

Elastic Stack Elasticsearch
1

My environment architecture is I have 15 server. The details as below

Group 1 (Standalone)

  • server 1 (2 app)
  • server 2 3 app)
  • server 3 (2 app)
  • server 4 (3 app)

Group 2 (master and slave)

  • server 5 master (2 app)
  • server 6 slave (2 app)

Group 3 (master and 2 slave)

  • server 7 master (3 app)
  • server 8 slave (3 app)
  • server 9 slave (3 app)

Group 4 (1 master 5 slave)

  • server 10 master (1 app)
  • server 11 slave (1 app)
  • server 12 slave (1 app)
  • server 13 slave (1 app)
  • server 14 slave (1 app)
  • server 15 slave (1 app)

Each application have 15 -20 logs

What is the best way in creating the index in logstash?

Is below better for index pattern
app_name-log_name-YYMMDD

Later i want to visual in Kibana in below form table panel form consist of time, message based on respective log name

The below is data for a sinngle log. other log have simillar pattern but different data. i want to dispaly only for single log show single table

image
image1999×684 128 KB

2

this will explode in number of index. too many if you do it daily.
you have to find out how large each index is per application if they are small then you might want to do it monthly index per application if you want to seperate by application

if you want to do it by server then you have to check that.

basically it is bad idea to have too many small index.

3

hi creating multiple index should be faster right in performance than creating single index? When i want search for specific data in 1 logs. If single index, in that index will have alot of logs but i jsut want to search for a particular logs. wont it be slow?

4

yes it does. I thought along the same logic when I was first learning elk. but as you will grow. too many index will slow down everything as cluster have to keep stat of everything. and there is some hard limit of how many shard each server can have. once you reach that limit you will have many other problem.

once I started following rule that most used index will be max 10gig, less use index I use 30gig or more, since then I have very less problem and speed is not an issue so far.

5

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.