Separate shards


(Amin) #1

Hi all
I have design this structure for my enterprise network with more than 280 server :

Logstash(1)===>Docker (Redis)=====>Logstash(2)=======>Elasticsearch(Clustre)

my config is as follow :

logstash1 :

input {
beats {
port => "5044"
type => "win-log-test"
}
syslogt {
port => "5514"
type => "syslog"
}
}
output {
redis{
host => "172.24.36.20"
data_type =>"list"
key => "sadra"
}
}

Logstash2

input {
redis {
host => "172.24.36.20"
type => "redis-input"
data_type => "list"
key => "sadra"
}
}
output {
elasticsearch {
hosts => [ "172.24.36.27" ]
index => "dnswinlog-%{+YYYY.MM.dd}"
}
}

now I collect log from more than 200 windows but I need to send each window's Log to separate shard ( or anything like shard maybe I am mistake in shard ) in elastic
now This system collect log from all windows and store them in one daily shard but I want to put them in seperate shard and seperate day.
thanks


(Mark Walkom) #2

I think you may have posted this a little too early.


(Amin) #3

Hello
I update my text please see It
thanks


(Mark Walkom) #4

I think you mean you want to send things to different indexes?


(Amin) #5

yes please


(Christian Dahlqvist) #6

Why do you want to do this? Having lots of small shards is inefficient and a waste of resources as each shard has some overhead..


(Amin) #7

I need solution to visualize them in kibana in real time and show them to special depatments like NOC and SOC


(Christian Dahlqvist) #8

That can be done even if they are all in one index by applying filters.


(Amin) #9

ok would you please tell me how write filter and where wirte this filter ? in kibana or in Logstash ?


(Christian Dahlqvist) #10

Even if you have the data in a separate index, you can create and save filtered searches in Kibana and build visualisations on these.


(system) #11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.