# Bulk indexing: single segment per shard

**URL:** <https://discuss.elastic.co/t/bulk-indexing-single-segment-per-shard/166152>\
**Category:** Elasticsearch\
**Created:** [January 29, 2019, 11:28am UTC](https://discuss.elastic.co/t/bulk-indexing-single-segment-per-shard/166152 "2019-01-29T11:28:07Z")\
**Posts on this page:** 5\
**Page:** 1

<div class="post-metadata">

**Author:** ![John16](https://avatars.discourse-cdn.com/v4/letter/j/ea666f/32.png) [@John16](https://discuss.elastic.co/u/John16)\
**Post date:** [January 29, 2019, 11:28am UTC](https://discuss.elastic.co/t/bulk-indexing-single-segment-per-shard/166152/1 "2019-01-29T11:28:07Z")

</div>

Hello,

I need to bulk index a large number of documents starting with empty index. I know I can do a (long-running) \_forcemerge later to reduce number of segments to 1, but I wonder if there is a settings so I can get an index with a single segment without this step?

I am indexing on a separate server and I want to get index fully optimized for fast searches.

Thanks!

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [January 29, 2019, 1:03pm UTC](https://discuss.elastic.co/t/bulk-indexing-single-segment-per-shard/166152/2 "2019-01-29T13:03:55Z")

</div>

> [@John16](#):
>
> I know I can do a (long-running) \_forcemerge later to reduce number of segments to 1, but I wonder if there is a settings so I can get an index with a single segment without this step?

Elasticsearch assumes that you're going to continue to index documents and therefore does not merge segments this enthusiastically. Over-merging (e.g. merging to a single segment on a shard that is still indexing documents) can cause performance issues, so Elasticsearch won't do it unless you specifically ask it to.

Why do you want to avoid a final `_forcemerge` after your indexing has finished?

---

<div class="post-metadata">

**Author:** ![John16](https://avatars.discourse-cdn.com/v4/letter/j/ea666f/32.png) [@John16](https://discuss.elastic.co/u/John16)\
**Post date:** [January 29, 2019, 1:17pm UTC](https://discuss.elastic.co/t/bulk-indexing-single-segment-per-shard/166152/3 "2019-01-29T13:17:58Z")

</div>

> [@DavidTurner](#):
>
> Why do you want to avoid a final `_forcemerge` after your indexing has finished?

Because it takes very long and I know from the very beginning that I am starting with empty index, add ~100 millions of docs and then use index in read-only manner.

I want to get single-segment index as fast as possible.

---

<div class="post-metadata">

**Author:** ![DavidTurner](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/davidturner/32/22453_2.png) [@DavidTurner](https://discuss.elastic.co/u/DavidTurner)\
**Post date:** [January 29, 2019, 1:36pm UTC](https://discuss.elastic.co/t/bulk-indexing-single-segment-per-shard/166152/4 "2019-01-29T13:36:44Z")

</div>

I know of no easy way around having Elasticsearch write its data in multiple segments and then merge them together later.

If your documents do not fit into memory (specifically the [indexing buffer](https://www.elastic.co/guide/en/elasticsearch/reference/current/indexing-buffer.html)) then Elasticsearch needs to write a segment each time this buffer fills up.

If your indexing generates a translog larger than [`index.translog.flush_threshold_size`](https://www.elastic.co/guide/en/elasticsearch/reference/6.5/index-modules-translog.html#_translog_settings) then Elasticsearch will perform a flush each time this threshold is reached.

You should be sure to follow [the instructions on tuning for indexing speed](https://www.elastic.co/guide/en/elasticsearch/reference/6.5/tune-for-indexing-speed.html) since this will help to generate fewer segments. Particularly, ensure that you are not refreshing too frequently.

Are you sure that the final force merge is actually worth it? You seem to be trying to optimise the process of building a shard from scratch, which implies that you will be doing it quite often. How much extra search performance does it buy you?

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [February 26, 2019, 1:36pm UTC](https://discuss.elastic.co/t/bulk-indexing-single-segment-per-shard/166152/5 "2019-02-26T13:36:48Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.