POST a bulk to elasticsearch from datatables output

Elastic Stack Elasticsearch
1

Hello, I just installed elasticsearch today. I downloaded some stock market data from an exchange using curl and saved them as a json. I'm having trouble with uploading the json files to the elasticsearch. I'm trying to use the bulk API but it always returns malformed action. Here are what my data looks like.

{"draw":0,"recordsTotal":510,"recordsFiltered":510,"data":[{"No":1,"IDStockSummary":1385663,"Date":"2015-01-05T00:00:00","StockCode":"AALI","StockName":"Astra Agro Lestari Tbk.","Remarks":"--S1BXO1","Previous":24575.0,"OpenPrice":24600.0,"FirstTrade":24600.0,"High":24700.0,"Low":24350.0,"Close":24675.0,"Change":100.0,"Volume":589900.0,"Value":14492460000.0,"Frequency":790.0,"IndexIndividual":1910.3,"Offer":24675.0,"OfferVolume":95300.0,"Bid":24650.0,"BidVolume":3500.0,"ListedShares":1574745000.0,"TradebleShares":1574745000.0,"WeightForIndex":1574745000.0,"ForeignSell":71600.0,"ForeignBuy":229700.0,"DelistingDate":"","NonRegularVolume":0.0,"NonRegularValue":0.0,"NonRegularFrequency":0.0,"persen":null,"percentage":null},{"No":2,"IDStockSummary":1385664,"Date":"2015-01-05T00:00:00","StockCode":"ABBA","StockName":"Mahaka Media Tbk.","Remarks":"--U9---2","Previous":62.0,"OpenPrice":0.0,"FirstTrade":0.0,"High":62.0,"Low":57.0,"Close":60.0,"Change":-2.0,"Volume":690900.0,"Value":40174700.0,"Frequency":63.0,"IndexIndividual":105.3,"Offer":60.0,"OfferVolume":49100.0,"Bid":59.0,"BidVolume":101300.0,"ListedShares":2755125000.0,"TradebleShares":2755125000.0,"WeightForIndex":2755125000.0,"ForeignSell":0.0,"ForeignBuy":0.0,"DelistingDate":"","NonRegularVolume":0.0,"NonRegularValue":0.0,"NonRegularFrequency":0.0,"persen":null,"percentage":null},

and so on to 500 company. Every single one of them is on a single line.
For the sake of readability, I'll put the pretty version below

{
  "draw": 0,
  "recordsTotal": 510,
  "recordsFiltered": 510,
  "data": [
    {
      "No": 1,
      "IDStockSummary": 1385663,
      "Date": "2015-01-05T00:00:00",
      "StockCode": "AALI",
      "StockName": "Astra Agro Lestari Tbk.",
      "Remarks": "--S1BXO1",
      "Previous": 24575,
      "OpenPrice": 24600,
      "FirstTrade": 24600,
      "High": 24700,
      "Low": 24350,
      "Close": 24675,
      "Change": 100,
      "Volume": 589900,
      "Value": 14492460000,
      "Frequency": 790,
      "IndexIndividual": 1910.3,
      "Offer": 24675,
      "OfferVolume": 95300,
      "Bid": 24650,
      "BidVolume": 3500,
      "ListedShares": 1574745000,
      "TradebleShares": 1574745000,
      "WeightForIndex": 1574745000,
      "ForeignSell": 71600,
      "ForeignBuy": 229700,
      "DelistingDate": "",
      "NonRegularVolume": 0,
      "NonRegularValue": 0,
      "NonRegularFrequency": 0,
      "persen": null,
      "percentage": null
    },
    {
      "No": 2,
      "IDStockSummary": 1385664,
      "Date": "2015-01-05T00:00:00",
      "StockCode": "ABBA",
      "StockName": "Mahaka Media Tbk.",
      "Remarks": "--U9---2",
      "Previous": 62,
      "OpenPrice": 0,
      "FirstTrade": 0,
      "High": 62,
      "Low": 57,
      "Close": 60,
      "Change": -2,
      "Volume": 690900,
      "Value": 40174700,
      "Frequency": 63,
      "IndexIndividual": 105.3,
      "Offer": 60,
      "OfferVolume": 49100,
      "Bid": 59,
      "BidVolume": 101300,
      "ListedShares": 2755125000,
      "TradebleShares": 2755125000,
      "WeightForIndex": 2755125000,
      "ForeignSell": 0,
      "ForeignBuy": 0,
      "DelistingDate": "",
      "NonRegularVolume": 0,
      "NonRegularValue": 0,
      "NonRegularFrequency": 0,
      "persen": null,
      "percentage": null
    },
//and so on
  ]
}

I've read the Bulk API documentation however it still doesn't work. I've also read that this is a DataTables server-side post Any idea how can I fix this? Thank you

2

Could you print the first 2 lines of your bulk request?

3

Doing it the straightforward way,

$ curl -XPOST 'localhost:9200/stock/_bulk' --data-binary @20150105.json
{"error":"Content-Type header [application/x-www-form-urlencoded] is not supported","status":406}

Adding a content type:

$ curl -XPOST 'localhost:9200/stock/_bulk' -H 'Content-Type: application/json' --data-binary @20150105.json
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"The bulk request must be terminated by a newline [\n]"}],"type":"illegal_argument_exception","reason":"The bulk request must be terminated by a newline [\n]"},"status":400}

Adding a single new line at the end of the code:

$ curl -XPOST 'localhost:9200/stock/_bulk' -H 'Content-Type: application/json' --data-binary @20150105.json
{"error":{"root_cause":[{"type":"illegal_argument_exception","reason":"Malformed action/metadata line [1], expected START_OBJECT or END_OBJECT but found [VALUE_NUMBER]"}],"type":"illegal_argument_exception","reason":"Malformed action/metadata line [1], expected START_OBJECT or END_OBJECT but found [VALUE_NUMBER]"},"status":400}

I'm sorry. I'm actually very new to javascripts since I didn't learn much about it.

4

What are the first 2 lines of 20150105.json please?

5

20150105.json only contains 2 line. A whole data, the 1st line is just like on the post I've written, and the 2nd line is just a newline

6

It does not conform with the bulk API format which requires a header and the document.

Read the bulk api documentation

7

Thank you for your response before. How would you suggest me to edit this kind of file in order for me to work with this kind of data?

8

Did you read the documentation? Is there something you don't understand in it?

9

I read the wrong documentations, sorry. So this is what I understand now. Please correct me if I'm wrong.

To POST a document, the document needs a metadata, for example _index , _type, types, and mapping. _id will be automatically generated unless we use PUT.

to POST a bulk, it is required to create a request body, which looks like this.

{ action: { metadata }}\n
{ request body }\n
{ action: { metadata }}\n
{ request body }\n

The action metadata has 4 actions, which are create, index, update, and delete.
So in order for me to POST my json file, I need to edit the json file in a bulk request body format.

Is there any recommendations for me to do this automatically? I tried to use logstash but I failed to install them, I'm waiting for it to support Java 11.

10

I'm using logstash with Java8 for now.
Works well.

11

I would like to use Java8 however Java8 will no longer be supported in January 2019.

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.