# ES takes too much time to index data

**URL:** <https://discuss.elastic.co/t/es-takes-too-much-time-to-index-data/8593>\
**Category:** Elasticsearch\
**Created:** [August 1, 2012, 11:06am UTC](https://discuss.elastic.co/t/es-takes-too-much-time-to-index-data/8593 "2012-08-01T11:06:56Z")\
**Posts on this page:** 9\
**Page:** 1

<div class="post-metadata">

**Author:** ![ian](https://avatars.discourse-cdn.com/v4/letter/i/bb73d2/32.png) [@ian](https://discuss.elastic.co/u/ian)\
**Post date:** [August 1, 2012, 11:06am UTC](https://discuss.elastic.co/t/es-takes-too-much-time-to-index-data/8593/1 "2012-08-01T11:06:56Z")

</div>

Hi

I am using java API to index data and my ES version is 0.19.8 .It worked  
fine and about 155000 data got indexed .  
After that ES taking too much time to index the data . Code is in following  
link .

> <https://gist.github.com/siminoushad/3225800>

Thanks

---

<div class="post-metadata">

**Author:** ![Ivan](https://avatars.discourse-cdn.com/v4/letter/i/df788c/32.png) [@Ivan](https://discuss.elastic.co/u/Ivan)\
**Post date:** [August 1, 2012, 4:33pm UTC](https://discuss.elastic.co/t/es-takes-too-much-time-to-index-data/8593/2 "2012-08-01T16:33:27Z")

</div>

You should use bulk indexing instead of indexing every individual  
document separately. You are also refreshing the index after every  
single index operation, which is time-consuming. Use bulk indexing and  
disable refreshing (set the interval to -1) during batch indexing.

--  
Ivan

On Wed, Aug 1, 2012 at 4:06 AM, Simi MA [simi.ma@algotree.com](mailto:simi.ma@algotree.com) wrote:

> Hi
> 
> I am using java API to index data and my ES version is 0.19.8 .It worked  
> fine and about 155000 data got indexed .  
> After that ES taking too much time to index the data . Code is in following  
> link .  
> [Elasticsearch client · GitHub](https://gist.github.com/3225800)
> 
> Thanks

---

<div class="post-metadata">

**Author:** ![vineeth\_mohan](https://avatars.discourse-cdn.com/v4/letter/v/bc79bd/32.png) [@vineeth\_mohan](https://discuss.elastic.co/u/vineeth_mohan)\
**Post date:** [August 3, 2012, 6:09am UTC](https://discuss.elastic.co/t/es-takes-too-much-time-to-index-data/8593/3 "2012-08-03T06:09:03Z")

</div>

Hello ,

The case here is that the feeds come one by one and we are looking for a  
real time indexing solution.  
So we cant use bulk here.  
Also it is observed that with the 1.5 L feeds in ES , it takes like 45  
minutes to index a single feed using transport portocall on port 9300  
But it only takes a second using the standard 9200 port using curl tool.

What do you feel is the reason for this ?

Thanks  
Vineeth

On Wed, Aug 1, 2012 at 10:03 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:

> You should use bulk indexing instead of indexing every individual  
> document separately. You are also refreshing the index after every  
> single index operation, which is time-consuming. Use bulk indexing and  
> disable refreshing (set the interval to -1) during batch indexing.
> 
> --  
> Ivan
> 
> On Wed, Aug 1, 2012 at 4:06 AM, Simi MA [simi.ma@algotree.com](mailto:simi.ma@algotree.com) wrote:
> 
> > Hi
> > 
> > I am using java API to index data and my ES version is 0.19.8 .It worked  
> > fine and about 155000 data got indexed .  
> > After that ES taking too much time to index the data . Code is in  
> > following  
> > link .  
> > [Elasticsearch client · GitHub](https://gist.github.com/3225800)
> > 
> > Thanks

---

<div class="post-metadata">

**Author:** ![Shaun\_Etherton](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shaun_etherton/32/2540_2.png) [@Shaun\_Etherton](https://discuss.elastic.co/u/Shaun_Etherton)\
**Post date:** [August 3, 2012, 6:57am UTC](https://discuss.elastic.co/t/es-takes-too-much-time-to-index-data/8593/4 "2012-08-03T06:57:18Z")

</div>

I thought 9300 was for ES nodes to  
Communicate with each other - should you be using 9200 instead?

--  
Shaun

On Friday, 3 August 2012 at 15:39, Vineeth Mohan wrote:

> Hello ,
> 
> The case here is that the feeds come one by one and we are looking for a real time indexing solution.  
> So we cant use bulk here.  
> Also it is observed that with the 1.5 L feeds in ES , it takes like 45 minutes to index a single feed using transport portocall on port 9300  
> But it only takes a second using the standard 9200 port using curl tool.
> 
> What do you feel is the reason for this ?
> 
> Thanks  
> Vineeth
> 
> On Wed, Aug 1, 2012 at 10:03 PM, Ivan Brusic \<[ivan@brusic.com](mailto:ivan@brusic.com) ([mailto:ivan@brusic.com](mailto:ivan@brusic.com))\> wrote:
> 
> > You should use bulk indexing instead of indexing every individual  
> > document separately. You are also refreshing the index after every  
> > single index operation, which is time-consuming. Use bulk indexing and  
> > disable refreshing (set the interval to -1) during batch indexing.
> > 
> > --  
> > Ivan
> > 
> > On Wed, Aug 1, 2012 at 4:06 AM, Simi MA \<[simi.ma@algotree.com](mailto:simi.ma@algotree.com) ([mailto:simi.ma@algotree.com](mailto:simi.ma@algotree.com))\> wrote:
> > 
> > > Hi
> > > 
> > > I am using java API to index data and my ES version is 0.19.8 .It worked  
> > > fine and about 155000 data got indexed .  
> > > After that ES taking too much time to index the data . Code is in following  
> > > link .  
> > > [Elasticsearch client · GitHub](https://gist.github.com/3225800)
> > > 
> > > Thanks

---

<div class="post-metadata">

**Author:** ![vineeth\_mohan](https://avatars.discourse-cdn.com/v4/letter/v/bc79bd/32.png) [@vineeth\_mohan](https://discuss.elastic.co/u/vineeth_mohan)\
**Post date:** [August 3, 2012, 9:12am UTC](https://discuss.elastic.co/t/es-takes-too-much-time-to-index-data/8593/5 "2012-08-03T09:12:05Z")

</div>

Its the port used for transport client -

> **[Elasticsearch Platform — Find real-time answers at scale](https://www.elastic.co)**
>
> Power insights and outcomes with the Elasticsearch Platform and AI. See into your data and find answers that matter with enterprise solutions designed to help you build, observe, and protect. Try Elasticsearch free today.

@Shay - Please let me know what you think about this.

Thanks  
Vineeth

On Fri, Aug 3, 2012 at 12:27 PM, Shaun Etherton [shaun.etherton@gmail.com](mailto:shaun.etherton@gmail.com)wrote:

> I thought 9300 was for ES nodes to  
> Communicate with each other - should you be using 9200 instead?
> 
> --  
> Shaun
> 
> On Friday, 3 August 2012 at 15:39, Vineeth Mohan wrote:
> 
> Hello ,
> 
> The case here is that the feeds come one by one and we are looking for a  
> real time indexing solution.  
> So we cant use bulk here.  
> Also it is observed that with the 1.5 L feeds in ES , it takes like 45  
> minutes to index a single feed using transport portocall on port 9300  
> But it only takes a second using the standard 9200 port using curl tool.
> 
> What do you feel is the reason for this ?
> 
> Thanks  
> Vineeth
> 
> On Wed, Aug 1, 2012 at 10:03 PM, Ivan Brusic [ivan@brusic.com](mailto:ivan@brusic.com) wrote:
> 
> You should use bulk indexing instead of indexing every individual  
> document separately. You are also refreshing the index after every  
> single index operation, which is time-consuming. Use bulk indexing and  
> disable refreshing (set the interval to -1) during batch indexing.
> 
> --  
> Ivan
> 
> On Wed, Aug 1, 2012 at 4:06 AM, Simi MA [simi.ma@algotree.com](mailto:simi.ma@algotree.com) wrote:
> 
> > Hi
> > 
> > I am using java API to index data and my ES version is 0.19.8 .It worked  
> > fine and about 155000 data got indexed .  
> > After that ES taking too much time to index the data . Code is in  
> > following  
> > link .  
> > [Elasticsearch client · GitHub](https://gist.github.com/3225800)
> > 
> > Thanks

---

<div class="post-metadata">

**Author:** ![Shaun\_Etherton](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/shaun_etherton/32/2540_2.png) [@Shaun\_Etherton](https://discuss.elastic.co/u/Shaun_Etherton)\
**Post date:** [August 3, 2012, 1:15pm UTC](https://discuss.elastic.co/t/es-takes-too-much-time-to-index-data/8593/6 "2012-08-03T13:15:49Z")

</div>

Oh, I see.

## Thanks

Shaun

On Friday, 3 August 2012 at 18:42, Vineeth Mohan wrote:

> Its the port used for transport client - [Elastic — The Search AI Company | Elastic](http://www.elasticsearch.org/guide/reference/java-api/client.html)
> 
> @Shay - Please let me know what you think about this.
> 
> Thanks  
> Vineeth
> 
> On Fri, Aug 3, 2012 at 12:27 PM, Shaun Etherton \<[shaun.etherton@gmail.com](mailto:shaun.etherton@gmail.com) ([mailto:shaun.etherton@gmail.com](mailto:shaun.etherton@gmail.com))\> wrote:
> 
> > I thought 9300 was for ES nodes to  
> > Communicate with each other - should you be using 9200 instead?
> > 
> > --  
> > Shaun
> > 
> > On Friday, 3 August 2012 at 15:39, Vineeth Mohan wrote:
> > 
> > > Hello ,
> > > 
> > > The case here is that the feeds come one by one and we are looking for a real time indexing solution.  
> > > So we cant use bulk here.  
> > > Also it is observed that with the 1.5 L feeds in ES , it takes like 45 minutes to index a single feed using transport portocall on port 9300  
> > > But it only takes a second using the standard 9200 port using curl tool.
> > > 
> > > What do you feel is the reason for this ?
> > > 
> > > Thanks  
> > > Vineeth
> > > 
> > > On Wed, Aug 1, 2012 at 10:03 PM, Ivan Brusic \<[ivan@brusic.com](mailto:ivan@brusic.com) ([mailto:ivan@brusic.com](mailto:ivan@brusic.com))\> wrote:
> > > 
> > > > You should use bulk indexing instead of indexing every individual  
> > > > document separately. You are also refreshing the index after every  
> > > > single index operation, which is time-consuming. Use bulk indexing and  
> > > > disable refreshing (set the interval to -1) during batch indexing.
> > > > 
> > > > --  
> > > > Ivan
> > > > 
> > > > On Wed, Aug 1, 2012 at 4:06 AM, Simi MA \<[simi.ma@algotree.com](mailto:simi.ma@algotree.com) ([mailto:simi.ma@algotree.com](mailto:simi.ma@algotree.com))\> wrote:
> > > > 
> > > > > Hi
> > > > > 
> > > > > I am using java API to index data and my ES version is 0.19.8 .It worked  
> > > > > fine and about 155000 data got indexed .  
> > > > > After that ES taking too much time to index the data . Code is in following  
> > > > > link .  
> > > > > [Elasticsearch client · GitHub](https://gist.github.com/3225800)
> > > > > 
> > > > > Thanks

---

<div class="post-metadata">

**Author:** ![Wes\_Plunk](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/wes_plunk/32/2446_2.png) [@Wes\_Plunk](https://discuss.elastic.co/u/Wes_Plunk)\
**Post date:** [August 9, 2012, 8:33am UTC](https://discuss.elastic.co/t/es-takes-too-much-time-to-index-data/8593/7 "2012-08-09T08:33:43Z")

</div>

I'm seeing similar results, if anyone has any suggestions that would be  
great

On Wednesday, August 1, 2012 6:06:56 AM UTC-5, Aami wrote:

> Hi
> 
> I am using java API to index data and my ES version is 0.19.8 .It worked  
> fine and about 155000 data got indexed .  
> After that ES taking too much time to index the data . Code is in  
> following link .  
> [Elasticsearch client · GitHub](https://gist.github.com/3225800)
> 
> Thanks

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [August 9, 2012, 8:44am UTC](https://discuss.elastic.co/t/es-takes-too-much-time-to-index-data/8593/8 "2012-08-09T08:44:57Z")

</div>

Removing refresh should help.  
Then, for bulk indexing, it's best to use bulk features.

See: [Elasticsearch Platform — Find real-time answers at scale | Elastic](http://www.elasticsearch.org/guide/reference/java-api/bulk.html)  
[http://www.elasticsearch.org/guide/reference/java-api/bulk.html](http://www.elasticsearch.org/guide/reference/java-api/bulk.html)

Hope this helps.

David.

Le 9 août 2012 à 10:33, Wes Plunk [wes@wesandemily.com](mailto:wes@wesandemily.com) a écrit :

> I'm seeing similar results, if anyone has any suggestions that would be great
> 
> On Wednesday, August 1, 2012 6:06:56 AM UTC-5, Aami wrote:
> 
> > > Hi
> > 
> > I am using java API to index data and my ES version is 0.19.8 .It worked  
> > fine and about 155000 data got indexed .  
> > After that ES taking too much time to index the data . Code is in  
> > following link .  
> > [Elasticsearch client · GitHub](https://gist.github.com/3225800)
> > 
> > Thanks
> > 
> > ```
> > <https://gist.github.com/3225800>
> > 
> > ```
> > 
> > >

[https://gist.github.com/3225800](https://gist.github.com/3225800)

--  
David Pilato  
[http://www.scrutmydocs.org/](http://www.scrutmydocs.org/)  
[http://dev.david.pilato.fr/](http://dev.david.pilato.fr/)  
Twitter : @dadoonet / @elasticsearchfr / @scrutmydocs

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 6, 2017, 3:16am UTC](https://discuss.elastic.co/t/es-takes-too-much-time-to-index-data/8593/9 "2017-07-06T03:16:59Z")

</div>