# Connection errors during indexing using goroutines

**URL:** <https://discuss.elastic.co/t/connection-errors-during-indexing-using-goroutines/301040>\
**Category:** Elasticsearch\
**Created:** [March 30, 2022, 7:24am UTC](https://discuss.elastic.co/t/connection-errors-during-indexing-using-goroutines/301040 "2022-03-30T07:24:56Z")\
**Posts on this page:** 6\
**Page:** 1

<div class="post-metadata">

**Author:** ![apo\_p9](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/apo_p9/32/103694_2.png) [@apo\_p9](https://discuss.elastic.co/u/apo_p9)\
**Post date:** [March 30, 2022, 7:24am UTC](https://discuss.elastic.co/t/connection-errors-during-indexing-using-goroutines/301040/1 "2022-03-30T07:24:56Z")

</div>

I have a syncing utility in Go, that moves data from MongoDB to ES. Here is the gist of it. Create workgroup, launch a goroutine listening on a channel, pull results from Mongo and publish to a channel which will index it to ES using goroutines.

```nohighlight
wg := &sync.WaitGroup{}
go func() {
	for {
		select {
		case data := <-resultFeed:
			go func() {
                defer wg.Done()
                pushToES(data)
            }()
		}
	}
}()
... pull from Mongo
for {
    wg.Add(1)
    resultFeed <- row
}
...
wg.Wait()

```

Using the official ES client or olivere, the issue is that when using a goroutine to push the results to ES I get connection errors like:

```auto
no available connection: no Elasticsearch node available
read tcp 127.0.0.1:52778->127.0.0.1:9200: read: connection reset by peer
write tcp 127.0.0.1:54947->127.0.0.1:9200: write: broken pipe

```

Those come intermittently as some results will succeed then some will produce an error.  
Both ES and Mongo are default local installations. There are no logs produced from ES.

If I dont use a goroutine there are no errors but its obviously much slower and I chose Go specifically for the ability to index concurrently. I haven't tried using bulk requests because it is not only the indexing thats happening so I'd like to keep it per document.

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [March 30, 2022, 8:01am UTC](https://discuss.elastic.co/t/connection-errors-during-indexing-using-goroutines/301040/2 "2022-03-30T08:01:46Z")

</div>

I have no idea about your code, so all of the following is just a crazy assumption, which you are free to dismiss.

When using go-routines or going async in general, are you capping the number of concurrent requests to Elasticsearch or do you keep sending new requests without making sure there is a concurrent limit (i.e. only 5 requests at the same time in flight and triggering the 6th one only, if another one returned)?

That could be a problem.

---

<div class="post-metadata">

**Author:** ![apo\_p9](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/apo_p9/32/103694_2.png) [@apo\_p9](https://discuss.elastic.co/u/apo_p9)\
**Post date:** [March 30, 2022, 8:12am UTC](https://discuss.elastic.co/t/connection-errors-during-indexing-using-goroutines/301040/3 "2022-03-30T08:12:40Z")

</div>

Interesting. There are no limitations right now.  
@spinscale Would there be a configuration setting that I can modify to increase the number of requests allowed in ES?

---

<div class="post-metadata">

**Author:** ![spinscale](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/spinscale/32/25011_2.png) [@spinscale](https://discuss.elastic.co/u/spinscale)\
**Post date:** [March 30, 2022, 8:47am UTC](https://discuss.elastic.co/t/connection-errors-during-indexing-using-goroutines/301040/4 "2022-03-30T08:47:31Z")

</div>

Doing this on the server side is always an arms race IMO. I'd rather solve this on the client side (and maybe have a simple test first, if this is really the problem, before going deeper). If Elasticsearch is overwhelmed with the number of index operations (but the HTTP connection still works), it will tell you.

Also, make sure to reuse your HTTP connections instead of creating new ones for each request.

---

<div class="post-metadata">

**Author:** ![apo\_p9](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/apo_p9/32/103694_2.png) [@apo\_p9](https://discuss.elastic.co/u/apo_p9)\
**Post date:** [March 30, 2022, 8:52am UTC](https://discuss.elastic.co/t/connection-errors-during-indexing-using-goroutines/301040/5 "2022-03-30T08:52:28Z")

</div>

@spinscale Thank you for your suggestion, I implemented a buffer and havent seen the errors since.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [April 27, 2022, 8:52am UTC](https://discuss.elastic.co/t/connection-errors-during-indexing-using-goroutines/301040/6 "2022-04-27T08:52:43Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.