# Separate shards

**URL:** <https://discuss.elastic.co/t/separate-shards/74781>\
**Category:** Logstash\
**Created:** [February 12, 2017, 4:47am UTC](https://discuss.elastic.co/t/separate-shards/74781 "2017-02-12T04:47:48Z")\
**Posts on this page:** 11\
**Page:** 1

<div class="post-metadata">

**Author:** ![Amin1](https://avatars.discourse-cdn.com/v4/letter/a/e47c2d/32.png) [@Amin1](https://discuss.elastic.co/u/Amin1)\
**Post date:** [February 12, 2017, 4:47am UTC](https://discuss.elastic.co/t/separate-shards/74781/1 "2017-02-12T04:47:48Z")

</div>

Hi all  
I have design this structure for my enterprise network with more than 280 server :

Logstash(1)===\>Docker (Redis)=====\>Logstash(2)=======\>Elasticsearch(Clustre)

my config is as follow :

## logstash1 :

## input { beats { port =\> "5044" type =\> "win-log-test" } syslogt { port =\> "5514" type =\> "syslog" } } output { redis{ host =\> "172.24.36.20" data\_type =\>"list" key =\> "sadra" } }

## Logstash2

## input { redis { host =\> "172.24.36.20" type =\> "redis-input" data\_type =\> "list" key =\> "sadra" } } output { elasticsearch { hosts =\> ["172.24.36.27"] index =\> "dnswinlog-%{+YYYY.MM.dd}" } }

now I collect log from more than 200 windows but I need to send each window's Log to separate shard ( or anything like shard maybe I am mistake in shard ) in elastic  
now This system collect log from all windows and store them in one daily shard but I want to put them in seperate shard and seperate day.  
thanks

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [February 12, 2017, 5:50am UTC](https://discuss.elastic.co/t/separate-shards/74781/2 "2017-02-12T05:50:55Z")

</div>

I think you may have posted this a little too early.

---

<div class="post-metadata">

**Author:** ![Amin1](https://avatars.discourse-cdn.com/v4/letter/a/e47c2d/32.png) [@Amin1](https://discuss.elastic.co/u/Amin1)\
**Post date:** [February 12, 2017, 6:33am UTC](https://discuss.elastic.co/t/separate-shards/74781/3 "2017-02-12T06:33:44Z")

</div>

Hello  
I update my text please see It  
thanks

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [February 12, 2017, 6:52am UTC](https://discuss.elastic.co/t/separate-shards/74781/4 "2017-02-12T06:52:26Z")

</div>

I think you mean you want to send things to different indexes?

---

<div class="post-metadata">

**Author:** ![Amin1](https://avatars.discourse-cdn.com/v4/letter/a/e47c2d/32.png) [@Amin1](https://discuss.elastic.co/u/Amin1)\
**Post date:** [February 12, 2017, 7:04am UTC](https://discuss.elastic.co/t/separate-shards/74781/5 "2017-02-12T07:04:10Z")

</div>

yes please

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [February 12, 2017, 7:23am UTC](https://discuss.elastic.co/t/separate-shards/74781/6 "2017-02-12T07:23:34Z")

</div>

Why do you want to do this? Having lots of small shards is inefficient and a waste of resources as each shard has some overhead..

---

<div class="post-metadata">

**Author:** ![Amin1](https://avatars.discourse-cdn.com/v4/letter/a/e47c2d/32.png) [@Amin1](https://discuss.elastic.co/u/Amin1)\
**Post date:** [February 12, 2017, 7:25am UTC](https://discuss.elastic.co/t/separate-shards/74781/7 "2017-02-12T07:25:41Z")

</div>

I need solution to visualize them in kibana in real time and show them to special depatments like NOC and SOC

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [February 12, 2017, 7:27am UTC](https://discuss.elastic.co/t/separate-shards/74781/8 "2017-02-12T07:27:32Z")

</div>

That can be done even if they are all in one index by applying filters.

---

<div class="post-metadata">

**Author:** ![Amin1](https://avatars.discourse-cdn.com/v4/letter/a/e47c2d/32.png) [@Amin1](https://discuss.elastic.co/u/Amin1)\
**Post date:** [February 12, 2017, 8:41am UTC](https://discuss.elastic.co/t/separate-shards/74781/9 "2017-02-12T08:41:23Z")

</div>

ok would you please tell me how write filter and where wirte this filter ? in kibana or in Logstash ?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [February 12, 2017, 10:20am UTC](https://discuss.elastic.co/t/separate-shards/74781/10 "2017-02-12T10:20:50Z")

</div>

Even if you have the data in a separate index, you can create and save [filtered searches](https://www.elastic.co/guide/en/kibana/current/field-filter.html) in Kibana and build visualisations on these.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [March 12, 2017, 10:21am UTC](https://discuss.elastic.co/t/separate-shards/74781/11 "2017-03-12T10:21:01Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.