# Logstash target on Elasticsearch Architecture **URL:** \ **Category:** Elasticsearch\ **Created:** [October 4, 2018, 1:19am UTC](https://discuss.elastic.co/t/logstash-target-on-elasticsearch-architecture/150952 "2018-10-04T01:19:52Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![negrote](https://avatars.discourse-cdn.com/v4/letter/n/c4cdca/32.png) [@negrote](https://discuss.elastic.co/u/negrote)\ **Post date:** [October 4, 2018, 1:19am UTC](https://discuss.elastic.co/t/logstash-target-on-elasticsearch-architecture/150952/1 "2018-10-04T01:19:52Z")
Hi, I have the following architecture to elasticsearch 3 dedicated master eligible nodes (8 GB RAM + 2 CPUs) 3 data/ingest node (64GB RAM + 8 CPUs) 1 coordinating node (24 GB RAM + 4 CPUs) Which the better approach? Configure logstash to send outputs to coordinating node or change data node to be data/master node and send logstash outputs to data/master nodes? The coordinating node will be receive Kibana requests. ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [October 4, 2018, 7:06am UTC](https://discuss.elastic.co/t/logstash-target-on-elasticsearch-architecture/150952/2 "2018-10-04T07:06:57Z")
Another option would be to keep the dedicated master nodes and configure Logstash to send data directly to the data/ingest nodes. If dedicated master nodes are not required and you change the layout, you could send data directly to the master/data/ingest nodes. ---
**Author:** ![negrote](https://avatars.discourse-cdn.com/v4/letter/n/c4cdca/32.png) [@negrote](https://discuss.elastic.co/u/negrote)\ **Post date:** [October 4, 2018, 2:51pm UTC](https://discuss.elastic.co/t/logstash-target-on-elasticsearch-architecture/150952/3 "2018-10-04T14:51:18Z")
Thanks Christian I think I can mantain the following architecture: 3 master deticated (Cluster management) 3 data/ingest (logstash send directly) 1 coordinating only (clients consumer) With this architecture I think that avoid a currently problem for duplication data when new index are create. Do you have any tip to avoid duplication data on index creation? Thanks ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [October 4, 2018, 3:00pm UTC](https://discuss.elastic.co/t/logstash-target-on-elasticsearch-architecture/150952/4 "2018-10-04T15:00:41Z")
> [@negrote](#): > > With this architecture I think that avoid a currently problem for duplication data when new index are create. > > Do you have any tip to avoid duplication data on index creation? I am not sure I understand what you are referring to. Can you please clarify? ---
**Author:** ![negrote](https://avatars.discourse-cdn.com/v4/letter/n/c4cdca/32.png) [@negrote](https://discuss.elastic.co/u/negrote)\ **Post date:** [October 4, 2018, 4:08pm UTC](https://discuss.elastic.co/t/logstash-target-on-elasticsearch-architecture/150952/5 "2018-10-04T16:08:28Z")
Sure!! I have a duplication data always that new index are created. My index are generated daily. I need to avoid duplication data on index creation. I think that my currently architecture do not support the the currently load of data and the bulk API receive several timeout. Below the error identified. [2018-06-17T20:00:47,039][DEBUG][o.e.a.a.i.m.p.TransportPutMappingAction] [cdv1prgrafapv03-node-1] failed to put mappings on indices [[[emm\_001-2018.06.18/y7YonDWiTo2AeED-mYxKFA]]], type [doc] org.elasticsearch.cluster.metadata.ProcessClusterEventTimeoutException: failed to process cluster event (put-mapping) within 30s at org.elasticsearch.cluster.service.MasterService$Batcher.lambda$null$0(MasterService.java:122) ~[elasticsearch-6.1.0.jar:6.1.0] at java.util.ArrayList.forEach(ArrayList.java:1249) ~[?:1.8.0\_65] at org.elasticsearch.cluster.service.MasterService$Batcher.lambda$onTimeout$1(MasterService.java:121) ~[elasticsearch-6.1.0.jar:6.1.0] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingRunnable.run(ThreadContext.java:568) [elasticsearch-6.1.0.jar:6.1.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) [?:1.8.0\_65] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) [?:1.8.0\_65] at java.lang.Thread.run(Thread.java:745) [?:1.8.0\_65] ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [October 4, 2018, 4:44pm UTC](https://discuss.elastic.co/t/logstash-target-on-elasticsearch-architecture/150952/6 "2018-10-04T16:44:22Z")
It seems like cluster updates are taking a really long time, which is causing problems. How many indices and shards do you have in your cluster? ---
**Author:** ![negrote](https://avatars.discourse-cdn.com/v4/letter/n/c4cdca/32.png) [@negrote](https://discuss.elastic.co/u/negrote)\ **Post date:** [October 4, 2018, 6:30pm UTC](https://discuss.elastic.co/t/logstash-target-on-elasticsearch-architecture/150952/7 "2018-10-04T18:30:28Z")
I have 24.572 shards and 2.458 indeces ---
**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\ **Post date:** [October 4, 2018, 6:36pm UTC](https://discuss.elastic.co/t/logstash-target-on-elasticsearch-architecture/150952/8 "2018-10-04T18:36:34Z")
That is far too many indices and shards for a cluster that size. Please read [this blog post on shards and sharding](https://www.elastic.co/blog/how-many-shards-should-i-have-in-my-elasticsearch-cluster) for some practical guidelines and then change you you shard your data so you reduce this number dramatically. ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [November 1, 2018, 6:36pm UTC](https://discuss.elastic.co/t/logstash-target-on-elasticsearch-architecture/150952/9 "2018-11-01T18:36:46Z")
This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.