# Help configuring a multi-process ES setup **URL:** \ **Category:** Elasticsearch\ **Created:** [August 8, 2012, 4:35pm UTC](https://discuss.elastic.co/t/help-configuring-a-multi-process-es-setup/8682 "2012-08-08T16:35:22Z")\ **Posts on this page:** 9\ **Page:** 1
**Author:** ![Jesus\_Sanz\_Marcos](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jesus_sanz_marcos/32/2779_2.png) [@Jesus\_Sanz\_Marcos](https://discuss.elastic.co/u/Jesus_Sanz_Marcos)\ **Post date:** [August 8, 2012, 4:35pm UTC](https://discuss.elastic.co/t/help-configuring-a-multi-process-es-setup/8682/1 "2012-08-08T16:35:22Z")
Hi, I am trying to build a ES based service. The idea is to have 16 ES processes in the same machine (one per core) that do not communicate with each other. I will redirect the PUT inserts to the 16 servers using the first letter of the md5 of {index/type/id} so I can have 16 independent ingestion processes. I will take care of the mapReduce operation in an upper layer. The question is how must I configure each single ES service so they don't speak with each other and they don't share/shard the documents. Thank you in advance and best regards, Jesús ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [August 8, 2012, 4:54pm UTC](https://discuss.elastic.co/t/help-configuring-a-multi-process-es-setup/8682/2 "2012-08-08T16:54:28Z")
You can disable multicast. See comments in elasticsearch.yml file. HTH David 😉 Twitter : @dadoonet / @elasticsearchfr Le 8 août 2012 à 18:35, Jesús Sanz Marcos [jesus@estudiocotacero.es](mailto:jesus@estudiocotacero.es) a écrit : > Hi, > > I am trying to build a ES based service. The idea is to have 16 ES processes in the same machine (one per core) that do not communicate with each other. > > I will redirect the PUT inserts to the 16 servers using the first letter of the md5 of {index/type/id} so I can have 16 independent ingestion processes. > > I will take care of the mapReduce operation in an upper layer. > > The question is how must I configure each single ES service so they don't speak with each other and they don't share/shard the documents. > > Thank you in advance and best regards, > > Jesús ---
**Author:** ![olof](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/olof/32/2721_2.png) [@olof](https://discuss.elastic.co/u/olof)\ **Post date:** [August 9, 2012, 7:48am UTC](https://discuss.elastic.co/t/help-configuring-a-multi-process-es-setup/8682/3 "2012-08-09T07:48:08Z")
Couldn't he also give them different cluster names? Though I suppose disabling the discovery has the same effect. Den onsdagen den 8:e augusti 2012 kl. 18:54:28 UTC+2 skrev David Pilato: > You can disable multicast. See comments in elasticsearch.yml file. > > HTH > David 😉 > Twitter : @dadoonet / @elasticsearchfr > > Le 8 août 2012 à 18:35, Jesús Sanz Marcos \\> > a écrit : > > > Hi, > > > > I am trying to build a ES based service. The idea is to have 16 ES > > processes in the same machine (one per core) that do not communicate with > > each other. > > > > I will redirect the PUT inserts to the 16 servers using the first letter > > of the md5 of {index/type/id} so I can have 16 independent ingestion > > processes. > > > > I will take care of the mapReduce operation in an upper layer. > > > > The question is how must I configure each single ES service so they > > don't speak with each other and they don't share/shard the documents. > > > > Thank you in advance and best regards, > > > > Jesús ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [August 9, 2012, 8:23am UTC](https://discuss.elastic.co/t/help-configuring-a-multi-process-es-setup/8682/4 "2012-08-09T08:23:05Z")
Sure he can. But if you have on the same LAN different versions of ES, let's say 0.17.5 and 0.19.8, you will have often exception because nodes try to connect to each other but they can't. I have already seen it in the past so I often disable multicast and define all my nodes in the unicast config. My 2 cents David. Le 9 août 2012 à 09:48, olof [onnilsson@gmail.com](mailto:onnilsson@gmail.com) a écrit : > Couldn't he also give them different cluster names? Though I suppose disabling > the discovery has the same effect. > > Den onsdagen den 8:e augusti 2012 kl. 18:54:28 UTC+2 skrev David Pilato: > > > > You can disable multicast. See comments in elasticsearch.yml file. > > > > HTH > > David 😉 > > Twitter : @dadoonet / @elasticsearchfr > > > > Le 8 août 2012 à 18:35, Jesús Sanz Marcos \< je...@estudiocotacero.es\> a > > écrit : > > > > > Hi, > > > > > > I am trying to build a ES based service. The idea is to have 16 ES > > > processes in the same machine (one per core) that do not communicate > > > with each other. > > > > > > I will redirect the PUT inserts to the 16 servers using the first > > > letter of the md5 of {index/type/id} so I can have 16 independent > > > ingestion processes. > > > > > > I will take care of the mapReduce operation in an upper layer. > > > > > > The question is how must I configure each single ES service so they > > > don't speak with each other and they don't share/shard the documents. > > > > > > Thank you in advance and best regards, > > > > > > Jesús -- David Pilato [http://www.scrutmydocs.org/](http://www.scrutmydocs.org/) [http://dev.david.pilato.fr/](http://dev.david.pilato.fr/) Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs ---
**Author:** ![Jesus\_Sanz\_Marcos](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jesus_sanz_marcos/32/2779_2.png) [@Jesus\_Sanz\_Marcos](https://discuss.elastic.co/u/Jesus_Sanz_Marcos)\ **Post date:** [August 9, 2012, 10:07am UTC](https://discuss.elastic.co/t/help-configuring-a-multi-process-es-setup/8682/5 "2012-08-09T10:07:13Z")
Hi! Thanks for your answers. I have configured the instances with separate folders and binaries, with configuration file like: http.port: 90XX. /// from 9000 to 9015 index.number\_of\_shards: 1 discovery.zen.ping.multicast.enabled: false But I still find documents from other instances. So if I PUT a doc in instance at port 9010, I can find it when I POST a search to instance at port 9005, for example. So servers are still speaking to each other... Thanks again Jesús ---
**Author:** ![Jesus\_Sanz\_Marcos](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jesus_sanz_marcos/32/2779_2.png) [@Jesus\_Sanz\_Marcos](https://discuss.elastic.co/u/Jesus_Sanz_Marcos)\ **Post date:** [August 9, 2012, 10:07am UTC](https://discuss.elastic.co/t/help-configuring-a-multi-process-es-setup/8682/6 "2012-08-09T10:07:14Z")
Hi! Thanks for your answers. I have configured the instances with separate folders and binaries, with configuration file like: http.port: 90XX. /// from 9000 to 9015 index.number\_of\_shards: 1 discovery.zen.ping.multicast.enabled: false But I still find documents from other instances. So if I PUT a doc in instance at port 9010, I can find it when I POST a search to instance at port 9005, for example. So servers are still speaking to each other... Thanks again Jesús ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [August 9, 2012, 11:24am UTC](https://discuss.elastic.co/t/help-configuring-a-multi-process-es-setup/8682/7 "2012-08-09T11:24:44Z")
Did you delete all the previous documents and reindex everything? Le 9 août 2012 à 12:07, "Jesús Sanz Marcos" [jesus@estudiocotacero.es](mailto:jesus@estudiocotacero.es) a écrit : > Hi! > Thanks for your answers. I have configured the instances with separate folders > and binaries, with configuration file like: > > http.port: 90XX. /// from 9000 to 9015 > index.number\_of\_shards: 1 > discovery.zen.ping.multicast.enabled: false > > But I still find documents from other instances. So if I PUT a doc in instance > at port 9010, I can find it when I POST a search to instance at port 9005, for > example. So servers are still speaking to each other... > > Thanks again > > Jesús -- David Pilato [http://www.scrutmydocs.org/](http://www.scrutmydocs.org/) [http://dev.david.pilato.fr/](http://dev.david.pilato.fr/) Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs ---
**Author:** ![Jesus\_Sanz\_Marcos](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/jesus_sanz_marcos/32/2779_2.png) [@Jesus\_Sanz\_Marcos](https://discuss.elastic.co/u/Jesus_Sanz_Marcos)\ **Post date:** [August 9, 2012, 6:16pm UTC](https://discuss.elastic.co/t/help-configuring-a-multi-process-es-setup/8682/8 "2012-08-09T18:16:17Z")
Hi, I had removed the data folders but I did re-install all the instances from the tar.gz file using the following configuration. > http.port: 9013 cluster.name: elasticsearch\_9013 index.number\_of\_shards: 1 discovery.zen.ping.multicast.enabled: false And now the documents are isolated in each instance. THANKS The EC2 machine has 8 cores and I am using 16 ES instances. Each instance is receiving PUT requests from a separate c++ application running in the same machine. It hardly gets to 25% of CPU. Do you think that the I/O is the bottleneck here or I could send more requests in parallel to each instance. Have you ever seen ES at 100% during ingestion? Thanks a lot! ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 3:16am UTC](https://discuss.elastic.co/t/help-configuring-a-multi-process-es-setup/8682/9 "2017-07-06T03:16:54Z")