Separate shards

Elastic Stack Logstash
1

Hi all
I have design this structure for my enterprise network with more than 280 server :

Logstash(1)===>Docker (Redis)=====>Logstash(2)=======>Elasticsearch(Clustre)

my config is as follow :

logstash1 :

input {
beats {
port => "5044"
type => "win-log-test"
}
syslogt {
port => "5514"
type => "syslog"
}
}
output {
redis{
host => "172.24.36.20"
data_type =>"list"
key => "sadra"
}
}

Logstash2

input {
redis {
host => "172.24.36.20"
type => "redis-input"
data_type => "list"
key => "sadra"
}
}
output {
elasticsearch {
hosts => [ "172.24.36.27" ]
index => "dnswinlog-%{+YYYY.MM.dd}"
}
}

now I collect log from more than 200 windows but I need to send each window's Log to separate shard ( or anything like shard maybe I am mistake in shard ) in elastic
now This system collect log from all windows and store them in one daily shard but I want to put them in seperate shard and seperate day.
thanks

2

I think you may have posted this a little too early.

3

Hello
I update my text please see It
thanks

4

I think you mean you want to send things to different indexes?

5

yes please

6

Why do you want to do this? Having lots of small shards is inefficient and a waste of resources as each shard has some overhead..

7

I need solution to visualize them in kibana in real time and show them to special depatments like NOC and SOC

8

That can be done even if they are all in one index by applying filters.

9

ok would you please tell me how write filter and where wirte this filter ? in kibana or in Logstash ?

10

Even if you have the data in a separate index, you can create and save filtered searches in Kibana and build visualisations on these.

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.