# Co-locate elasticsearch with hadoop/yarn **URL:** \ **Category:** Elasticsearch\ **Created:** [September 4, 2014, 6:32pm UTC](https://discuss.elastic.co/t/co-locate-elasticsearch-with-hadoop-yarn/19630 "2014-09-04T18:32:41Z")\ **Posts on this page:** 5\ **Page:** 1
**Author:** ![Ronny\_Vaningh\_2](https://avatars.discourse-cdn.com/v4/letter/r/f1d935/32.png) [@Ronny\_Vaningh\_2](https://discuss.elastic.co/u/Ronny_Vaningh_2)\ **Post date:** [September 4, 2014, 6:32pm UTC](https://discuss.elastic.co/t/co-locate-elasticsearch-with-hadoop-yarn/19630/1 "2014-09-04T18:32:41Z")
Hi I have some beefy boxes with 512 Gb ram and I would like to co-locate yarn/hadoop with elasticsearch Does anyone have experience in doing the same ? How did you split the resources (memory/disk) across both functions ? Hdfs like jbod while E.S. like raid10 Thank Reg -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CACA5U5mW8zn0BLWryeePdR%2B0yiBGhd\_9pQWVy-Uufj3H-oeQhA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CACA5U5mW8zn0BLWryeePdR%2B0yiBGhd_9pQWVy-Uufj3H-oeQhA%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![andrew](https://avatars.discourse-cdn.com/v4/letter/a/f19dbf/32.png) [@andrew](https://discuss.elastic.co/u/andrew)\ **Post date:** [September 5, 2014, 12:30am UTC](https://discuss.elastic.co/t/co-locate-elasticsearch-with-hadoop-yarn/19630/2 "2014-09-05T00:30:35Z")
I have not tried this but my initial thoughts would be - Set ES\_HEAP\_SIZE = 30 GB, give Hadoop an appropriate amount, leave the rest for the OS cache. - Set the filesystem paths where ES and Hadoop store data to separate physical disk(s). You don't want them contending for bandwidth. - You don't have to use RAID for ES, you can use multiple data paths if you have multiple disks. - At this size, many people choose to run multiple instances of ES on a single physical. Give each instance 30GB and point to different disks. A On Sep 4, 2014, at 11:32 AM, Ronny Vaningh [ronny@guard-it.be](mailto:ronny@guard-it.be) wrote: > Hi > > I have some beefy boxes with 512 Gb ram and I would like to co-locate yarn/hadoop with elasticsearch > > Does anyone have experience in doing the same ? > How did you split the resources (memory/disk) across both functions ? > > Hdfs like jbod while E.S. like raid10 > > Thank > > Reg > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CACA5U5mW8zn0BLWryeePdR%2B0yiBGhd\_9pQWVy-Uufj3H-oeQhA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CACA5U5mW8zn0BLWryeePdR%2B0yiBGhd_9pQWVy-Uufj3H-oeQhA%40mail.gmail.com). > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/0A8CF809-7F38-40B2-AE23-508F1477B823%40elasticsearch.com](https://groups.google.com/d/msgid/elasticsearch/0A8CF809-7F38-40B2-AE23-508F1477B823%40elasticsearch.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![Ronny\_Vaningh\_2](https://avatars.discourse-cdn.com/v4/letter/r/f1d935/32.png) [@Ronny\_Vaningh\_2](https://discuss.elastic.co/u/Ronny_Vaningh_2)\ **Post date:** [September 5, 2014, 6:19am UTC](https://discuss.elastic.co/t/co-locate-elasticsearch-with-hadoop-yarn/19630/3 "2014-09-05T06:19:01Z")
Thanks Andrew If I'm correct e.s. will first fill up the first disk, and so on if you specify multiple data paths vs scattering over them like hdfs.. correct? You overcome this by giving easch e.s. instanve a data path on a sepparate disk Thanks Regards Ronny On 5 Sep 2014 02:30, "Andrew Selden" [andrew.selden@elasticsearch.com](mailto:andrew.selden@elasticsearch.com) wrote: > I have not tried this but my initial thoughts would be > > - Set ES\_HEAP\_SIZE = 30 GB, give Hadoop an appropriate amount, leave the > rest for the OS cache. > - Set the filesystem paths where ES and Hadoop store data to separate > physical disk(s). You don’t want them contending for bandwidth. > - You don’t have to use RAID for ES, you can use multiple data paths if > you have multiple disks. > - At this size, many people choose to run multiple instances of ES on a > single physical. Give each instance 30GB and point to different disks. > > A > > On Sep 4, 2014, at 11:32 AM, Ronny Vaningh [ronny@guard-it.be](mailto:ronny@guard-it.be) wrote: > > Hi > > I have some beefy boxes with 512 Gb ram and I would like to co-locate > yarn/hadoop with elasticsearch > > Does anyone have experience in doing the same ? > How did you split the resources (memory/disk) across both functions ? > > Hdfs like jbod while E.S. like raid10 > > Thank > > Reg > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/CACA5U5mW8zn0BLWryeePdR%2B0yiBGhd\_9pQWVy-Uufj3H-oeQhA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CACA5U5mW8zn0BLWryeePdR%2B0yiBGhd_9pQWVy-Uufj3H-oeQhA%40mail.gmail.com) > [https://groups.google.com/d/msgid/elasticsearch/CACA5U5mW8zn0BLWryeePdR%2B0yiBGhd\_9pQWVy-Uufj3H-oeQhA%40mail.gmail.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/CACA5U5mW8zn0BLWryeePdR%2B0yiBGhd_9pQWVy-Uufj3H-oeQhA%40mail.gmail.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to the Google Groups > "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit > [https://groups.google.com/d/msgid/elasticsearch/0A8CF809-7F38-40B2-AE23-508F1477B823%40elasticsearch.com](https://groups.google.com/d/msgid/elasticsearch/0A8CF809-7F38-40B2-AE23-508F1477B823%40elasticsearch.com) > [https://groups.google.com/d/msgid/elasticsearch/0A8CF809-7F38-40B2-AE23-508F1477B823%40elasticsearch.com?utm\_medium=email&utm\_source=footer](https://groups.google.com/d/msgid/elasticsearch/0A8CF809-7F38-40B2-AE23-508F1477B823%40elasticsearch.com?utm_medium=email&utm_source=footer) > . > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CACA5U5%3DUD\_HuLL\_DEM4J-kKL330pvnex5QLjwJkF-f\_ac1Sehg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CACA5U5%3DUD_HuLL_DEM4J-kKL330pvnex5QLjwJkF-f_ac1Sehg%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\ **Post date:** [September 5, 2014, 6:40am UTC](https://discuss.elastic.co/t/co-locate-elasticsearch-with-hadoop-yarn/19630/4 "2014-09-05T06:40:23Z")
f you give 2 disks to Elasticsearch, it will fill both at the same time. Not one after the other. -- David 😉 Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs Le 5 sept. 2014 à 08:19, Ronny Vaningh [ronny@guard-it.be](mailto:ronny@guard-it.be) a écrit : Thanks Andrew If I'm correct e.s. will first fill up the first disk, and so on if you specify multiple data paths vs scattering over them like hdfs.. correct? You overcome this by giving easch e.s. instanve a data path on a sepparate disk Thanks Regards Ronny > On 5 Sep 2014 02:30, "Andrew Selden" [andrew.selden@elasticsearch.com](mailto:andrew.selden@elasticsearch.com) wrote: > > I have not tried this but my initial thoughts would be > > - Set ES\_HEAP\_SIZE = 30 GB, give Hadoop an appropriate amount, leave the rest for the OS cache. > - Set the filesystem paths where ES and Hadoop store data to separate physical disk(s). You don’t want them contending for bandwidth. > - You don’t have to use RAID for ES, you can use multiple data paths if you have multiple disks. > - At this size, many people choose to run multiple instances of ES on a single physical. Give each instance 30GB and point to different disks. > > A > > > On Sep 4, 2014, at 11:32 AM, Ronny Vaningh [ronny@guard-it.be](mailto:ronny@guard-it.be) wrote: > > > > Hi > > > > I have some beefy boxes with 512 Gb ram and I would like to co-locate yarn/hadoop with elasticsearch > > > > Does anyone have experience in doing the same ? > > How did you split the resources (memory/disk) across both functions ? > > > > Hdfs like jbod while E.S. like raid10 > > > > Thank > > > > Reg > > > > -- > > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CACA5U5mW8zn0BLWryeePdR%2B0yiBGhd\_9pQWVy-Uufj3H-oeQhA%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CACA5U5mW8zn0BLWryeePdR%2B0yiBGhd_9pQWVy-Uufj3H-oeQhA%40mail.gmail.com). > > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). > > -- > You received this message because you are subscribed to the Google Groups "elasticsearch" group. > To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). > To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/0A8CF809-7F38-40B2-AE23-508F1477B823%40elasticsearch.com](https://groups.google.com/d/msgid/elasticsearch/0A8CF809-7F38-40B2-AE23-508F1477B823%40elasticsearch.com). > For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/CACA5U5%3DUD\_HuLL\_DEM4J-kKL330pvnex5QLjwJkF-f\_ac1Sehg%40mail.gmail.com](https://groups.google.com/d/msgid/elasticsearch/CACA5U5%3DUD_HuLL_DEM4J-kKL330pvnex5QLjwJkF-f_ac1Sehg%40mail.gmail.com). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). -- You received this message because you are subscribed to the Google Groups "elasticsearch" group. To unsubscribe from this group and stop receiving emails from it, send an email to [elasticsearch+unsubscribe@googlegroups.com](mailto:elasticsearch+unsubscribe@googlegroups.com). To view this discussion on the web visit [https://groups.google.com/d/msgid/elasticsearch/2B91F6B4-1A18-46FF-97DA-14CD55B63D51%40pilato.fr](https://groups.google.com/d/msgid/elasticsearch/2B91F6B4-1A18-46FF-97DA-14CD55B63D51%40pilato.fr). For more options, visit [https://groups.google.com/d/optout](https://groups.google.com/d/optout). ---
**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\ **Post date:** [July 6, 2017, 1:04am UTC](https://discuss.elastic.co/t/co-locate-elasticsearch-with-hadoop-yarn/19630/5 "2017-07-06T01:04:11Z")
Closed on July 6, 2017, 1:04am UTC