# Advice on Scaling writes

**URL:** <https://discuss.elastic.co/t/advice-on-scaling-writes/278518>\
**Category:** Elasticsearch\
**Created:** [July 13, 2021, 10:07am UTC](https://discuss.elastic.co/t/advice-on-scaling-writes/278518 "2021-07-13T10:07:28Z")\
**Posts on this page:** 4\
**Page:** 2

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [July 19, 2021, 2:33pm UTC](https://discuss.elastic.co/t/advice-on-scaling-writes/278518/21 "2021-07-19T14:33:48Z")

</div>

Nice!

Impressed with the 15K/s with 1 filebeat curious what were setting / config is.

For large ingest indices Primary Shards \<= Data Nodes, equal to for large indices (within reason). For Smaller you want just leave the defaults. For this case if you are running 2 nodes I would try 2 primaries, 3 nodes, 3 Primaries

---

<div class="post-metadata">

**Author:** ![Baygon](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/baygon/32/91792_2.png) [@Baygon](https://discuss.elastic.co/u/Baygon)\
**Post date:** [July 20, 2021, 3:08pm UTC](https://discuss.elastic.co/t/advice-on-scaling-writes/278518/22 "2021-07-20T15:08:56Z")

</div>

Thanks. I will have very large indices on the long run, but was thinking of using rollover to avoid having to scale the shards.

No specific setting other than the ones you suggested earlier in this post. Likely the bulk\_max\_size and queue.mem helped.  
Also the content I have is composed of many small files (from 3k to 4Mb maximum), so that might help with parallelisation.  
I'm currently using the CSV processor, but might shift to using the Winston logger for nodejs so I don't need a processor at all (which might increase perf a little as well).

---

<div class="post-metadata">

**Author:** ![stephenb](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/stephenb/32/40856_2.png) [@stephenb](https://discuss.elastic.co/u/stephenb)\
**Post date:** [July 20, 2021, 3:29pm UTC](https://discuss.elastic.co/t/advice-on-scaling-writes/278518/23 "2021-07-20T15:29:13Z")

</div>

Thanks,

> [@Baygon](#):
>
> Thanks. I will have very large indices on the long run, but was thinking of using rollover to avoid having to scale the shards.

2 shards provides more parallelism than a single shard if you have 2 nodes, that is the only point, if you get enough throughput with 1 shard that is fine.

You can use ILM.  
Set Share Size to 50GB  
Primaries to 2 or 1

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 17, 2021, 3:30pm UTC](https://discuss.elastic.co/t/advice-on-scaling-writes/278518/24 "2021-08-17T15:30:03Z")

</div>

Closed on August 17, 2021, 3:30pm UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.

[Previous page](https://discuss.elastic.co/t/advice-on-scaling-writes/278518.md?page=1)
