# WHY index stops accepting any documents

**URL:** <https://discuss.elastic.co/t/why-index-stops-accepting-any-documents/50825>\
**Category:** Elasticsearch\
**Created:** [May 24, 2016, 10:32am UTC](https://discuss.elastic.co/t/why-index-stops-accepting-any-documents/50825 "2016-05-24T10:32:22Z")\
**Posts on this page:** 17\
**Page:** 1

<div class="post-metadata">

**Author:** ![nskalis](https://avatars.discourse-cdn.com/v4/letter/n/ebca7d/32.png) [@nskalis](https://discuss.elastic.co/u/nskalis)\
**Post date:** [May 24, 2016, 10:32am UTC](https://discuss.elastic.co/t/why-index-stops-accepting-any-documents/50825/1 "2016-05-24T10:32:22Z")

</div>

Hi guys,

I would really need your advice on this one;

There is 1 elastic search node with 4 indices. kopf shows  
 ![](https://us1.discourse-cdn.com/elastic/original/2X/2/25d22950def30670d7d4d4cc492e91add7085ea9.png)

After 22h, the \_\_10 index stops indexing documents, while the other ones operate as expected

 ![](https://us1.discourse-cdn.com/elastic/original/2X/9/9df32cb8290c9df36ae5369782539f4b66468e90.png)

The data are getting indexed using bulk requests via HTTP and this the log when "not working"  
Responce is **200**

```
2016-05-24 12:26:36.910 : DEBUG : IPacc-datastore : bin_packing_cache : > process_id "31922768" [IPacc:cache] found data "IPacc:cache:__10:pmacct" and calculated "2" bins of size "150000" plus "41974" messages
2016-05-24 12:27:05.100 : DEBUG : IPacc-datastore : bin_packing_cache : > {'req': <Response [200]>, 'pid': 31922768, 'rc': True}
2016-05-24 12:27:05.631 : DEBUG : IPacc-datastore : bin_packing_cache : > {'req': <Response [200]>, 'pid': 31922768, 'rc': True}
2016-05-24 12:27:05.886 : DEBUG : IPacc-datastore : bin_packing_cache : > {'req': <Response [200]>, 'pid': 31922768, 'rc': True}

```

Could you please please help me and try to find what is going wrong ?

Nikos

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [May 24, 2016, 10:40am UTC](https://discuss.elastic.co/t/why-index-stops-accepting-any-documents/50825/2 "2016-05-24T10:40:04Z")

</div>

It looks like you are hitting the Lucene limit of 2 billion documents per shard. Overall the shards look very large, so I would recommend using a higher number of shards for each index. If you tell us a bit more about your use case we may be able to provide better advice.

---

<div class="post-metadata">

**Author:** ![nskalis](https://avatars.discourse-cdn.com/v4/letter/n/ebca7d/32.png) [@nskalis](https://discuss.elastic.co/u/nskalis)\
**Post date:** [May 24, 2016, 11:09am UTC](https://discuss.elastic.co/t/why-index-stops-accepting-any-documents/50825/3 "2016-05-24T11:09:59Z")

</div>

Thanks a lot @Christian_Dahlqvist  
I was not aware of that limit. And I reach that in ~22h.  
The objective is to hold data for ~1w or so.  
I have defined also a \_ttl equal to 7d.

I guess setting the number of shards to 15 would suffice ?  
Could you possibly advise on any other setting that may needs tweaking ?

Below are the settings and the mapping:

---

<div class="post-metadata">

**Author:** ![nskalis](https://avatars.discourse-cdn.com/v4/letter/n/ebca7d/32.png) [@nskalis](https://discuss.elastic.co/u/nskalis)\
**Post date:** [May 24, 2016, 11:10am UTC](https://discuss.elastic.co/t/why-index-stops-accepting-any-documents/50825/4 "2016-05-24T11:10:20Z")

</div>

> ```
> {
> "IPacc": {
> "_routing": {
> "required": true
> },
> "_ttl": {
> "default": 604800000,
> "enabled": true
> },
> "dynamic": "true",
> "_all": {
> "enabled": false
> },
> "properties": {
> "dev_name": {
> "index": "not_analyzed",
> "type": "string"
> },
> "dst_as": {
> "type": "integer"
> },
> "dst_mask": {
> "type": "integer"
> },
> "local_pref": {
> "type": "integer"
> },
> "time_stamp": {
> "format": "epoch_second",
> "type": "date"
> },
> "peer_dst_as": {
> "type": "integer"
> },
> "peer_src_as": {
> "type": "integer"
> },
> "src_mask": {
> "type": "integer"
> },
> "dst_ip": {
> "index": "not_analyzed",
> "type": "string"
> },
> "packets": {
> "type": "long"
> },
> "peer_src_ip": {
> "index": "not_analyzed",
> "type": "string"
> },
> "src_ip": {
> "index": "not_analyzed",
> "type": "string"
> },
> "peer_dst_ip": {
> "index": "not_analyzed",
> "type": "string"
> },
> "med_metric": {
> "type": "integer"
> },
> "bytes": {
> "type": "long"
> },
> "ifindex_in": {
> "type": "integer"
> },
> "ifindex_out": {
> "type": "integer"
> },
> "proto": {
> "index": "not_analyzed",
> "type": "string"
> },
> "tos": {
> "type": "integer"
> },
> "as_path": {
> "type": "string"
> },
> "bgp_comm": {
> "type": "string"
> },
> "src_as": {
> "type": "integer"
> }
> }
> },
> "_default_": {
> "_routing": {
> "required": true
> },
> "_ttl": {
> "default": 604800000,
> "enabled": true
> },
> "dynamic": "true",
> "_all": {
> "enabled": false
> },
> "properties": {
> "dev_name": {
> "index": "not_analyzed",
> "type": "string"
> },
> "dst_as": {
> "type": "integer"
> },
> "dst_mask": {
> "type": "integer"
> },
> "local_pref": {
> "type": "integer"
> },
> "time_stamp": {
> "format": "epoch_second",
> "type": "date"
> },
> "peer_dst_as": {
> "type": "integer"
> },
> "peer_src_as": {
> "type": "integer"
> },
> "src_mask": {
> "type": "integer"
> },
> "dst_ip": {
> "index": "not_analyzed",
> "type": "string"
> },
> "packets": {
> "type": "long"
> },
> "peer_src_ip": {
> "index": "not_analyzed",
> "type": "string"
> },
> "src_ip": {
> "index": "not_analyzed",
> "type": "string"
> },
> "peer_dst_ip": {
> "index": "not_analyzed",
> "type": "string"
> },
> "med_metric": {
> "type": "integer"
> },
> "bytes": {
> "type": "long"
> },
> "ifindex_in": {
> "type": "integer"
> },
> "ifindex_out": {
> "type": "integer"
> },
> "proto": {
> "index": "not_analyzed",
> "type": "string"
> },
> "tos": {
> "type": "integer"
> },
> "as_path": {
> "type": "string"
> },
> "bgp_comm": {
> "type": "string"
> },
> "src_as": {
> "type": "integer"
> }
> }
> }
> }
> 
> ```

---

<div class="post-metadata">

**Author:** ![nskalis](https://avatars.discourse-cdn.com/v4/letter/n/ebca7d/32.png) [@nskalis](https://discuss.elastic.co/u/nskalis)\
**Post date:** [May 24, 2016, 11:10am UTC](https://discuss.elastic.co/t/why-index-stops-accepting-any-documents/50825/5 "2016-05-24T11:10:27Z")

</div>

> ```
> {
> "index": {
> "cache": {
> "field": {
> "type": "soft"
> },
> "filter": {
> "type": "node"
> },
> "query": {
> "enable": "true"
> }
> },
> "memory": {
> "index_buffer_size": "20%"
> },
> "refresh_interval": "1m",
> "fielddata": {
> "cache": {
> "type": "node"
> }
> },
> "number_of_shards": "1",
> "merge": {
> "scheduler": {
> "type": "concurrent"
> },
> "async": "true",
> "policy": {
> "type": "log_byte_size",
> "merge_factor": "15"
> }
> },
> "creation_date": "1463945724550",
> "store": {
> "throttle": {
> "type": "none"
> }
> },
> "number_of_replicas": "0",
> "uuid": "qbzWKQW4TMyKq_H1AirNXQ",
> "version": {
> "created": "2030299"
> }
> }
> }
> 
> ```

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [May 24, 2016, 11:16am UTC](https://discuss.elastic.co/t/why-index-stops-accepting-any-documents/50825/6 "2016-05-24T11:16:07Z")

</div>

You may want to look into using [time-based indices](https://www.elastic.co/guide/en/elasticsearch/guide/2.x/time-based.html#index-per-timeframe) instead as this is significantly more efficient than using TTL (which is being deprecated).

---

<div class="post-metadata">

**Author:** ![nskalis](https://avatars.discourse-cdn.com/v4/letter/n/ebca7d/32.png) [@nskalis](https://discuss.elastic.co/u/nskalis)\
**Post date:** [May 24, 2016, 11:22am UTC](https://discuss.elastic.co/t/why-index-stops-accepting-any-documents/50825/7 "2016-05-24T11:22:12Z")

</div>

Thanks 🙂 . Indeed I read the guideline. But time-based indices do not 'fit' well in my data pipeline because of the bulk insertion. According to the documentation they will replace it with another mechanism (retention policy) instead of the \_ttl.

Taking advantage of your prompt replies;  
May you have any suggestions in regards to any shards settings, etc ? That need to be setup in a better way ?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [May 24, 2016, 11:26am UTC](https://discuss.elastic.co/t/why-index-stops-accepting-any-documents/50825/8 "2016-05-24T11:26:23Z")

</div>

I do not see how bulk inserts are related to whether you use time based indices or not? Could you please elaborate?

---

<div class="post-metadata">

**Author:** ![nskalis](https://avatars.discourse-cdn.com/v4/letter/n/ebca7d/32.png) [@nskalis](https://discuss.elastic.co/u/nskalis)\
**Post date:** [May 24, 2016, 11:34am UTC](https://discuss.elastic.co/t/why-index-stops-accepting-any-documents/50825/9 "2016-05-24T11:34:23Z")

</div>

Because in order to post documents in a time-based index, the index has to be created first.  
I am not using log stash that can create the index automatically.  
And not all of the docs when I do a bulk request have the same timestamp.  
I would like to avoid to set up indices before hand with specific dates.  
Or there is another way of doing it ?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [May 24, 2016, 11:39am UTC](https://discuss.elastic.co/t/why-index-stops-accepting-any-documents/50825/10 "2016-05-24T11:39:40Z")

</div>

Indices are created when you first write data to them, so you do not need to explicitly create them. Logstash uploads an index template that specifies how it is to be created and then just writes to the correct index pattern. If the index does not exist, it is created automatically based on the template.

---

<div class="post-metadata">

**Author:** ![nskalis](https://avatars.discourse-cdn.com/v4/letter/n/ebca7d/32.png) [@nskalis](https://discuss.elastic.co/u/nskalis)\
**Post date:** [May 24, 2016, 11:58am UTC](https://discuss.elastic.co/t/why-index-stops-accepting-any-documents/50825/11 "2016-05-24T11:58:38Z")

</div>

But how do you set the mapping of the data ? I know that you can do that also per request. But is more efficient if I create the index and just post the data. In my case, on every index there is 1 mapping.  
So what will happen when is close to midnight and there is a bulk of 150k docs to be pushed ?  
The only option is to group them by date before i do post, and if the mapping is not there to create it, correct ?

---

<div class="post-metadata">

**Author:** ![Christian\_Dahlqvist](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/christian_dahlqvist/32/4617_2.png) [@Christian\_Dahlqvist](https://discuss.elastic.co/u/Christian_Dahlqvist)\
**Post date:** [May 24, 2016, 12:05pm UTC](https://discuss.elastic.co/t/why-index-stops-accepting-any-documents/50825/12 "2016-05-24T12:05:36Z")

</div>

The [index template](https://www.elastic.co/guide/en/elasticsearch/reference/2.3/indices-templates.html#delete) contains the mapping as well as the index settings. If you send a bulk request inserting documents that do not already exist, these will be automatically created according to the index template, so you do not need to group the data and batch it up based on timestamp.

This is what the [default logstash index template](https://github.com/logstash-plugins/logstash-output-elasticsearch/blob/master/lib/logstash/outputs/elasticsearch/elasticsearch-template.json) looks like, and as you can see it contains index settings as well as mappings to be used.

---

<div class="post-metadata">

**Author:** ![nskalis](https://avatars.discourse-cdn.com/v4/letter/n/ebca7d/32.png) [@nskalis](https://discuss.elastic.co/u/nskalis)\
**Post date:** [May 24, 2016, 12:16pm UTC](https://discuss.elastic.co/t/why-index-stops-accepting-any-documents/50825/13 "2016-05-24T12:16:57Z")

</div>

Am sorry am not getting it.. 😐

Practically speaking, let's say that I have an index named _xxx10_  
And I would like to post documents using time-based indices.  
How the request would look like ? What will be the name of the index ?

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [May 29, 2016, 1:01pm UTC](https://discuss.elastic.co/t/why-index-stops-accepting-any-documents/50825/14 "2016-05-29T13:01:19Z")

</div>

> Practically speaking, let's say that I have an index named xxx10  
> And I would like to post documents using time-based indices.  
> How the request would look like ? What will be the name of the index ?

Your indexing request would post the data to an index named something like xxx10-YYYY.MM.DD (e.g. xxx10-2016.05.29). If that index doesn't exist it'll be created. If there's an index template that matches the name of the index it'll get applied. You'd have to figure out the date to put in the index name.

---

<div class="post-metadata">

**Author:** ![nskalis](https://avatars.discourse-cdn.com/v4/letter/n/ebca7d/32.png) [@nskalis](https://discuss.elastic.co/u/nskalis)\
**Post date:** [May 29, 2016, 2:41pm UTC](https://discuss.elastic.co/t/why-index-stops-accepting-any-documents/50825/15 "2016-05-29T14:41:57Z")

</div>

thanks a lot @magnusbaeck i had the same impression as well. so  
either i have to create the index before hand  
or make the aggregate post by separating the data based on timestamp and create the index if necessary.

---

<div class="post-metadata">

**Author:** ![magnusbaeck](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/magnusbaeck/32/44943_2.png) [@magnusbaeck](https://discuss.elastic.co/u/magnusbaeck)\
**Post date:** [May 29, 2016, 3:14pm UTC](https://discuss.elastic.co/t/why-index-stops-accepting-any-documents/50825/16 "2016-05-29T15:14:05Z")

</div>

No, I still don't think you're getting it. You _never_ have to explicitly create the index. All you have to do is change the index name in your requests (bulk or not) to include the current date.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 5, 2017, 10:48pm UTC](https://discuss.elastic.co/t/why-index-stops-accepting-any-documents/50825/17 "2017-07-05T22:48:16Z")

</div>