Separate shards

(Amin) #1

Hi all
I have design this structure for my enterprise network with more than 280 server :

Logstash(1)===>Docker (Redis)=====>Logstash(2)=======>Elasticsearch(Clustre)

my config is as follow :

logstash1 :

input {
beats {
port => "5044"
type => "win-log-test"
syslogt {
port => "5514"
type => "syslog"
output {
host => ""
data_type =>"list"
key => "sadra"


input {
redis {
host => ""
type => "redis-input"
data_type => "list"
key => "sadra"
output {
elasticsearch {
hosts => [ "" ]
index => "dnswinlog-%{+YYYY.MM.dd}"

now I collect log from more than 200 windows but I need to send each window's Log to separate shard ( or anything like shard maybe I am mistake in shard ) in elastic
now This system collect log from all windows and store them in one daily shard but I want to put them in seperate shard and seperate day.

(Mark Walkom) #2

I think you may have posted this a little too early.

(Amin) #3

I update my text please see It

(Mark Walkom) #4

I think you mean you want to send things to different indexes?

(Amin) #5

yes please

(Christian Dahlqvist) #6

Why do you want to do this? Having lots of small shards is inefficient and a waste of resources as each shard has some overhead..

(Amin) #7

I need solution to visualize them in kibana in real time and show them to special depatments like NOC and SOC

(Christian Dahlqvist) #8

That can be done even if they are all in one index by applying filters.

(Amin) #9

ok would you please tell me how write filter and where wirte this filter ? in kibana or in Logstash ?

(Christian Dahlqvist) #10

Even if you have the data in a separate index, you can create and save filtered searches in Kibana and build visualisations on these.

(system) #11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.