# POST a bulk to elasticsearch from datatables output

**URL:** <https://discuss.elastic.co/t/post-a-bulk-to-elasticsearch-from-datatables-output/161628>\
**Category:** Elasticsearch\
**Created:** [December 20, 2018, 6:29am UTC](https://discuss.elastic.co/t/post-a-bulk-to-elasticsearch-from-datatables-output/161628 "2018-12-20T06:29:55Z")\
**Posts on this page:** 12\
**Page:** 1

<div class="post-metadata">

**Author:** ![prajogo.atmaja](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/prajogo.atmaja/32/134170_2.png) [@prajogo.atmaja](https://discuss.elastic.co/u/prajogo.atmaja)\
**Post date:** [December 20, 2018, 6:29am UTC](https://discuss.elastic.co/t/post-a-bulk-to-elasticsearch-from-datatables-output/161628/1 "2018-12-20T06:29:55Z")

</div>

Hello, I just installed elasticsearch today. I downloaded some stock market data from an exchange using curl and saved them as a json. I'm having trouble with uploading the json files to the elasticsearch. I'm trying to use the bulk API but it always returns malformed action. Here are what my data looks like.

```
{"draw":0,"recordsTotal":510,"recordsFiltered":510,"data":[{"No":1,"IDStockSummary":1385663,"Date":"2015-01-05T00:00:00","StockCode":"AALI","StockName":"Astra Agro Lestari Tbk.","Remarks":"--S1BXO1","Previous":24575.0,"OpenPrice":24600.0,"FirstTrade":24600.0,"High":24700.0,"Low":24350.0,"Close":24675.0,"Change":100.0,"Volume":589900.0,"Value":14492460000.0,"Frequency":790.0,"IndexIndividual":1910.3,"Offer":24675.0,"OfferVolume":95300.0,"Bid":24650.0,"BidVolume":3500.0,"ListedShares":1574745000.0,"TradebleShares":1574745000.0,"WeightForIndex":1574745000.0,"ForeignSell":71600.0,"ForeignBuy":229700.0,"DelistingDate":"","NonRegularVolume":0.0,"NonRegularValue":0.0,"NonRegularFrequency":0.0,"persen":null,"percentage":null},{"No":2,"IDStockSummary":1385664,"Date":"2015-01-05T00:00:00","StockCode":"ABBA","StockName":"Mahaka Media Tbk.","Remarks":"--U9---2","Previous":62.0,"OpenPrice":0.0,"FirstTrade":0.0,"High":62.0,"Low":57.0,"Close":60.0,"Change":-2.0,"Volume":690900.0,"Value":40174700.0,"Frequency":63.0,"IndexIndividual":105.3,"Offer":60.0,"OfferVolume":49100.0,"Bid":59.0,"BidVolume":101300.0,"ListedShares":2755125000.0,"TradebleShares":2755125000.0,"WeightForIndex":2755125000.0,"ForeignSell":0.0,"ForeignBuy":0.0,"DelistingDate":"","NonRegularVolume":0.0,"NonRegularValue":0.0,"NonRegularFrequency":0.0,"persen":null,"percentage":null},

```

and so on to 500 company. Every single one of them is on a single line.  
For the sake of readability, I'll put the pretty version below

> ```
> {
> "draw": 0,
> "recordsTotal": 510,
> "recordsFiltered": 510,
> "data": [
> {
> "No": 1,
> "IDStockSummary": 1385663,
> "Date": "2015-01-05T00:00:00",
> "StockCode": "AALI",
> "StockName": "Astra Agro Lestari Tbk.",
> "Remarks": "--S1BXO1",
> "Previous": 24575,
> "OpenPrice": 24600,
> "FirstTrade": 24600,
> "High": 24700,
> "Low": 24350,
> "Close": 24675,
> "Change": 100,
> "Volume": 589900,
> "Value": 14492460000,
> "Frequency": 790,
> "IndexIndividual": 1910.3,
> "Offer": 24675,
> "OfferVolume": 95300,
> "Bid": 24650,
> "BidVolume": 3500,
> "ListedShares": 1574745000,
> "TradebleShares": 1574745000,
> "WeightForIndex": 1574745000,
> "ForeignSell": 71600,
> "ForeignBuy": 229700,
> "DelistingDate": "",
> "NonRegularVolume": 0,
> "NonRegularValue": 0,
> "NonRegularFrequency": 0,
> "persen": null,
> "percentage": null
> },
> {
> "No": 2,
> "IDStockSummary": 1385664,
> "Date": "2015-01-05T00:00:00",
> "StockCode": "ABBA",
> "StockName": "Mahaka Media Tbk.",
> "Remarks": "--U9---2",
> "Previous": 62,
> "OpenPrice": 0,
> "FirstTrade": 0,
> "High": 62,
> "Low": 57,
> "Close": 60,
> "Change": -2,
> "Volume": 690900,
> "Value": 40174700,
> "Frequency": 63,
> "IndexIndividual": 105.3,
> "Offer": 60,
> "OfferVolume": 49100,
> "Bid": 59,
> "BidVolume": 101300,
> "ListedShares": 2755125000,
> "TradebleShares": 2755125000,
> "WeightForIndex": 2755125000,
> "ForeignSell": 0,
> "ForeignBuy": 0,
> "DelistingDate": "",
> "NonRegularVolume": 0,
> "NonRegularValue": 0,
> "NonRegularFrequency": 0,
> "persen": null,
> "percentage": null
> },
> //and so on
> ]
> }
> 
> ```

I've read the [Bulk API documentation](https://www.elastic.co/guide/en/elasticsearch/reference/current/docs-bulk.html) however it still doesn't work. I've also read that this is a [DataTables server-side post](https://datatables.net/examples/server_side/post.html) Any idea how can I fix this? Thank you

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [December 20, 2018, 8:34am UTC](https://discuss.elastic.co/t/post-a-bulk-to-elasticsearch-from-datatables-output/161628/2 "2018-12-20T08:34:45Z")

</div>

Could you print the first 2 lines of your bulk request?

---

<div class="post-metadata">

**Author:** ![prajogo.atmaja](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/prajogo.atmaja/32/134170_2.png) [@prajogo.atmaja](https://discuss.elastic.co/u/prajogo.atmaja)\
**Post date:** [December 20, 2018, 5:16pm UTC](https://discuss.elastic.co/t/post-a-bulk-to-elasticsearch-from-datatables-output/161628/3 "2018-12-20T17:16:40Z")

</div>

Doing it the straightforward way,

> **$ curl -XPOST 'localhost:9200/stock/\_bulk' --data-binary @20150105.json**  
> {"error":"Content-Type header [application/x-www-form-urlencoded] is not supported","status":406}

Adding a content type:

> **$ curl -XPOST 'localhost:9200/stock/\_bulk' -H 'Content-Type: application/json' --data-binary @20150105.json**  
> {"error":{"root\_cause":[{"type":"illegal\_argument\_exception","reason":"The bulk request must be terminated by a newline [\n]"}],"type":"illegal\_argument\_exception","reason":"The bulk request must be terminated by a newline [\n]"},"status":400}

Adding a single new line at the end of the code:

> **$ curl -XPOST 'localhost:9200/stock/\_bulk' -H 'Content-Type: application/json' --data-binary @20150105.json**  
> {"error":{"root\_cause":[{"type":"illegal\_argument\_exception","reason":"Malformed action/metadata line [1], expected START\_OBJECT or END\_OBJECT but found [VALUE\_NUMBER]"}],"type":"illegal\_argument\_exception","reason":"Malformed action/metadata line [1], expected START\_OBJECT or END\_OBJECT but found [VALUE\_NUMBER]"},"status":400}

I'm sorry. I'm actually very new to javascripts since I didn't learn much about it.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [December 20, 2018, 5:59pm UTC](https://discuss.elastic.co/t/post-a-bulk-to-elasticsearch-from-datatables-output/161628/4 "2018-12-20T17:59:24Z")

</div>

What are the first 2 lines of **20150105.json** please?

---

<div class="post-metadata">

**Author:** ![prajogo.atmaja](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/prajogo.atmaja/32/134170_2.png) [@prajogo.atmaja](https://discuss.elastic.co/u/prajogo.atmaja)\
**Post date:** [December 20, 2018, 6:25pm UTC](https://discuss.elastic.co/t/post-a-bulk-to-elasticsearch-from-datatables-output/161628/5 "2018-12-20T18:25:53Z")

</div>

**20150105.json** only contains 2 line. A whole data, the 1st line is just like on the post I've written, and the 2nd line is just a newline

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [December 20, 2018, 7:12pm UTC](https://discuss.elastic.co/t/post-a-bulk-to-elasticsearch-from-datatables-output/161628/6 "2018-12-20T19:12:34Z")

</div>

It does not conform with the bulk API format which requires a header and the document.

Read the bulk api documentation

---

<div class="post-metadata">

**Author:** ![prajogo.atmaja](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/prajogo.atmaja/32/134170_2.png) [@prajogo.atmaja](https://discuss.elastic.co/u/prajogo.atmaja)\
**Post date:** [December 20, 2018, 7:16pm UTC](https://discuss.elastic.co/t/post-a-bulk-to-elasticsearch-from-datatables-output/161628/7 "2018-12-20T19:16:00Z")

</div>

Thank you for your response before. How would you suggest me to edit this kind of file in order for me to work with this kind of data?

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [December 20, 2018, 7:45pm UTC](https://discuss.elastic.co/t/post-a-bulk-to-elasticsearch-from-datatables-output/161628/8 "2018-12-20T19:45:29Z")

</div>

Did you read the documentation? Is there something you don't understand in it?

---

<div class="post-metadata">

**Author:** ![prajogo.atmaja](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/prajogo.atmaja/32/134170_2.png) [@prajogo.atmaja](https://discuss.elastic.co/u/prajogo.atmaja)\
**Post date:** [December 20, 2018, 8:21pm UTC](https://discuss.elastic.co/t/post-a-bulk-to-elasticsearch-from-datatables-output/161628/9 "2018-12-20T20:21:29Z")

</div>

I read the wrong documentations, sorry. So this is what I understand now. Please correct me if I'm wrong.

To `POST` a document, the document needs a metadata, for example `_index` , `_type`, `types`, and `mapping`. `_id` will be automatically generated unless we use PUT.

to `POST` a `bulk`, it is required to create a request body, which looks like this.

> { action: { metadata }}\n  
> { request body }\n  
> { action: { metadata }}\n  
> { request body }\n

The action metadata has 4 actions, which are `create`, `index`, `update`, and `delete`.  
So in order for me to POST my json file, I need to edit the json file in a `bulk` request body format.

Is there any recommendations for me to do this automatically? I tried to use logstash but I failed to install them, I'm waiting for it to support Java 11.

---

<div class="post-metadata">

**Author:** ![dadoonet](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/dadoonet/32/137187_2.png) [@dadoonet](https://discuss.elastic.co/u/dadoonet)\
**Post date:** [December 20, 2018, 8:41pm UTC](https://discuss.elastic.co/t/post-a-bulk-to-elasticsearch-from-datatables-output/161628/10 "2018-12-20T20:41:12Z")

</div>

I'm using logstash with Java8 for now.  
Works well.

---

<div class="post-metadata">

**Author:** ![prajogo.atmaja](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/prajogo.atmaja/32/134170_2.png) [@prajogo.atmaja](https://discuss.elastic.co/u/prajogo.atmaja)\
**Post date:** [December 20, 2018, 9:04pm UTC](https://discuss.elastic.co/t/post-a-bulk-to-elasticsearch-from-datatables-output/161628/11 "2018-12-20T21:04:56Z")

</div>

I would like to use Java8 however Java8 will no longer be supported in January 2019.

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [January 17, 2019, 9:05pm UTC](https://discuss.elastic.co/t/post-a-bulk-to-elasticsearch-from-datatables-output/161628/12 "2019-01-17T21:05:00Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.