Problems with bulk indexing via JavaScript client

Elastic Stack Elasticsearch
1

Hi, I'm using a variant of the code for bulk indexing from the documentation for Elastic 6.x, and I'm having a few problems.

Here's a typical document:

{ index: { _index: 'messages', _type: '_doc', _id: 21653 } },
  { user_id: 11,
    creation: '2016-11-07T15:43:03:000',
    modification: '2019-04-28T11:40:16:000',
    title:
     'Dark Workflows: How 5 Signatures Became a Productivity Write-Off',
    note: 'REMOVED',
    links_to_asset: [ [Object] ],
    number_of_words: 804,
    phrases: [],
    favourite: [],
    message_id: 13,
    from: 'Wayne Smallman <REMOVED>',
    seen: 'unread',
    to_read: [],
    engagement: [],
    in_folder_id: 587 }

... and then it gets so far in and I get the following error:

{ took: 20,
errors: true,
items:
[ { index: [Object] },
{ index: [Object] },
{ index: [Object] },
{ index: [Object] },
{ index: [Object] },
{ index: [Object] },
{ index: [Object] },
{ index: [Object] },
{ index: [Object] },
{ index: [Object] } ] }

Let's face it, as errors go, that's not in the least bit helpful, nor do I understand where the data has gotten to for each of the corresponding indexes, assuming the data should have been listed there.

You'll have noticed links_to_asset: [ [Object] ], and I've tried converting each of the raw objects with JSON.stringify() versions, but that makes no difference.

Looking at the data, and after having added lots of tracer comments, it's choking on the first attempt at a bulk index, although — as explained — the data appears to be correct.

Some guidance would be much appreciated.

2

I'm not sure if this constitutes a degree of success or not, but after a few changes I'm not getting the unhelpful error.

async function runBulkInsert () {
  const { body: bulkResponse } = await client.bulk({
    refresh: true,
    body: [
      indexForBulk,
      assetAttributes
    ]
  })
  if (bulkResponse.errors) {
    console.log("Elastic:createIndicesReader:runBulkInsert:bulkResponse.errors", bulkResponse)
    process.exit(1)
  }
}
runBulkInsert().catch(error => {
  console.log("Elastic:createIndicesReader:runBulkInsert:catch", error.meta.body.error)
})

Here, indexForBulk and assetAttributes correspond to the index and document.

Regardless of the lack of errors, the bulk import still isn't working (I've checked with Postman and Kibana).

I've spent the entire weekend on this, and the data I'm sending to bulk() is consistent with the documentation, so I'm at a stage where I've exceeded the entire range of permutations of code I know.

3

I tried something different:

axios.post(`localhost:9200/_bulk`, parameters)
.then(function (response) {
  // handle success
  console.log("Elastic:axios:success", response);
})
.catch(function (error) {
  // handle error
  console.log("Elastic:axios:error", error);
})

... but now I'm getting another error:

{ Error: Request failed with status code 400
at createError (/REMOVED/node_modules/axios/lib/core/createError.js:16:15)
at settle (/REMOVED/node_modules/axios/lib/core/settle.js:17:12)
at IncomingMessage.handleStreamEnd (/REMOVED/node_modules/axios/lib/adapters/http.js:237:11)
at IncomingMessage.emit (events.js:187:15)
at endReadableNT (_stream_readable.js:1092:12)
at process._tickCallback (internal/process/next_tick.js:63:19)
config:
{ url: 'http://localhost:9200/_bulk',
method: 'post',
data: ...

4

Hello!
It looks like that the date format you are using is not correct:
Can you try to change it from 2016-11-07T15:43:03:000 to 2016-11-07T15:43:03.000 (the last column should be a dot).
If the issue persists I recommend you to run a single index operation, so you can easily see the error that elasticsearch is sending back to you.
If you can't fix the issue after the steps above, can you open a new issue in the JavaScript client and use this repository to reproduce your issue?
It would be very useful if you also write what you are getting and the expected behavior.
Thanks!

5

Hi! In the end, that was one of several issues, but I wouldn't have known until I made a change to the bulk error checking:

if (responseForBulkIndex.errors) {
  console.log("Elastic:createIndicesReader:runBulkInsert:responseForBulkIndex.errors")
  responseForBulkIndex.items.forEach(function(error) {
    console.log("Elastic:createIndicesReader:runBulkInsert:responseForBulkIndex.error", error)
  })
  process.exit(1)
}

Here, I'm looping over the [object] items to get at what was inside them, which was a stream of errors.

It would be useful to have Elastic surface these errors instead of burying them.

6

I'm sorry that you encountered some issues for reading the errors of the bulk API.
We'll work on improving the code examples and also provide a bulk helper in the future!

1 Like
7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.