# ElasticSearch Bulk Indexing Azure VM

**URL:** <https://discuss.elastic.co/t/elasticsearch-bulk-indexing-azure-vm/249846>\
**Category:** Elasticsearch\
**Created:** [September 24, 2020, 3:11pm UTC](https://discuss.elastic.co/t/elasticsearch-bulk-indexing-azure-vm/249846 "2020-09-24T15:11:12Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![irtzasuhail](https://avatars.discourse-cdn.com/v4/letter/i/a4c791/32.png) [@irtzasuhail](https://discuss.elastic.co/u/irtzasuhail)\
**Post date:** [September 24, 2020, 3:11pm UTC](https://discuss.elastic.co/t/elasticsearch-bulk-indexing-azure-vm/249846/1 "2020-09-24T15:11:13Z")

</div>

We are using a VM in Azure to run ElasticSearch to do some bulk indexing on our data. The biggest input JSON we've tested with so far is around 70 MB in size, and each time it fails with the node error 134 which translates locally to an Out of Memory Error. We tried increasing the heap size to 5 GB but to no avail. Any advice on how we can proceed?

```auto
 const { body: bulkResponse } = await client.bulk({
        refresh: true,
        body,
    });

```

This is the segment that fails each time. Is there any way that we can break down this larger JSON and iteratively bulk index those smaller files and collate the results at the end?

Update:

Found Bulk Helpers to break down larger documents, but it ends the call itself, rather than returning the index for processing further along.

```auto
    const result = await client.helpers.bulk({
        datasource: body,
        onDocument (doc) {
          return {
            index: { _index: 'my-index' },
          }
        },
        onDrop (doc) {
            console.log(doc)
          },
          retries: 3,
          refreshOnCompletion: true
      })

```

The helper sometimes returns this warning

```auto
 Warning: Unexpected call to 'log' on the context object after function execution has completed. Please check for asynchronous calls that are not awaited or calls to 'done' made before function execution completes.

```

---

<div class="post-metadata">

**Author:** ![warkolm](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/warkolm/32/39224_2.png) [@warkolm](https://discuss.elastic.co/u/warkolm)\
**Post date:** [September 28, 2020, 3:42am UTC](https://discuss.elastic.co/t/elasticsearch-bulk-indexing-azure-vm/249846/2 "2020-09-28T03:42:55Z")

</div>

What do the Elasticsearch logs show at this time?

---

<div class="post-metadata">

**Author:** ![irtzasuhail](https://avatars.discourse-cdn.com/v4/letter/i/a4c791/32.png) [@irtzasuhail](https://discuss.elastic.co/u/irtzasuhail)\
**Post date:** [October 12, 2020, 3:42pm UTC](https://discuss.elastic.co/t/elasticsearch-bulk-indexing-azure-vm/249846/3 "2020-10-12T15:42:24Z")

</div>

Hi,

Really sorry for the late response. I got side tracked onto other things.

Im testing it right now, and it times out at 30 minutes without sending back any response.

Elastic Logs show the following

```auto
{"type": "server", "timestamp": "2020-10-12T14:50:01,079Z", "level": "INFO", "component": "o.e.c.m.MetadataMappingService", "cluster.name": "talos-elasticsearch", "node.name": "talosnode-01", "message": "[words/xUmNWjA3RUahEqfrY1NCZQ] create_mapping [_doc]", "cluster.uuid": "t7Iu5owsRjGp0VoDKudeSA", "node.id": "ApGtWNeiQZiEKJcwbQ4ejA" }`

```

```
[2020-10-12T14:50:00,825][INFO][o.e.c.m.MetadataCreateIndexService] [talosnode-01] [words] creating index, cause [auto(bulk api)], templates [], shards [1]/[1]
[2020-10-12T14:50:01,079][INFO][o.e.c.m.MetadataMappingService] [talosnode-01] [words/xUmNWjA3RUahEqfrY1NCZQ] create_mapping [_doc]

```

```auto

Node stats says 

```

```
{
  "_nodes" : {
    "total" : 1,
    "successful" : 1,
    "failed" : 0
  },

```

```auto

```

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [November 9, 2020, 3:42pm UTC](https://discuss.elastic.co/t/elasticsearch-bulk-indexing-azure-vm/249846/4 "2020-11-09T15:42:29Z")

</div>

Closed on November 9, 2020, 3:42pm UTC

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.