How to insert data from a json file using bulk API in ES2.1

Elastic Stack Elasticsearch
1

Hi Experts,

I have downloaded accounts.json from here and saved it in my local drive . Path is E:\data\accounts.json

Now when I fired
POST /bank/account/_bulk?pretty --data-binary E:\data\accounts.json
I am getting below error
{
"error": {
"root_cause": [
{
"type": "parse_exception",
"reason": "Failed to derive xcontent"
}
],
"type": "parse_exception",
"reason": "Failed to derive xcontent"
},
"status": 400
}

Please suggest what needs to be done here . I believe bulk command is not able to pick the file path.

2

accounts ? shakespeare ?

I'm lost

3

oops my mistake actually i was trying to insert data for both ,that is account and shakespeare.

For now I want to learn it for accounts , I download it from here and saved it my local drive i.e E:/data/accounts.json.

My question is how I tell bulk API that the above json file is at what path ?

4

i use this code

$j = array(
"association_key" => $k['id'],
"text" => $k['text'],
"photo" => $k['photo_tweet']);
$j = json_encode($j);
$json_doc.='
{ "create" : { "_index" : "'.$index.'", "_type" : "'.$doc_type.'", "_id" : "0" }}
'.$j.'';
$baseUri = 'http://' . $search_host . ':' . $search_port . '/_bulk';
$ci = curl_init();
curl_setopt($ci, CURLOPT_URL, $baseUri);
curl_setopt($ci, CURLOPT_PORT, $search_port);
curl_setopt($ci, CURLOPT_TIMEOUT, 200);
curl_setopt($ci, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ci, CURLOPT_FORBID_REUSE, 0);
curl_setopt($ci, CURLOPT_CUSTOMREQUEST, 'PUT');
curl_setopt($ci, CURLOPT_POSTFIELDS, $json_doc);
curl_exec($ci);
curl_close($ci);

5

Thanks Ryodo,

But how to run this code ? Can't we just do directly the way it explained in the ES Documentation
https://www.elastic.co/guide/en/kibana/current/getting-started.html.

I am just struggling to tell ES the path of json file

6

@David

I Edited my question .

7

Are you doing that from SENSE? If so it won't work.

8

yes I am doing it from SENSE , is there any other way to do this ?

9

You can not use curl specific parameters in SENSE.

You can copy and paste the lines of your bulk file just after the _bulk command.
Or you can use curl on the command line.

10

HI Vikas
Can you please explain me how you got around the issue. Im doing same tutotrial..shakespear...and im am getting same error with Sense

11

@Mag_Carl, So as David suggested I used Curl rather Sense to fire bulk API.

12

{"index":{"_id":"58242012e1bc2218ea17f19d"}}
{"Vertical_id":32,
"Vertical_Name":"Ad Servers","Tech_ID":6603,
"Title":"Smart AdServer (Ad Servers)",
"Technology_Name":"Smart AdServer",
"Patterns":{
"Type":"File","Pattern":"cdn1.smartadserver.com/diff,smartadserver.com/call\n",
"text_content":"http://cdn1.smartadserver.com/diff/js/smartadserver.js"\u003e\u003c/script\u003e /javascript", src="http://ww619.smartadserver.com/call/pubj/32872/(home)/13955/m/799""}
}}

i have this type of document, and i am using bulk api like this
curl -XPOST 'localhost:9200/test/test_tech/_bulk?pretty' --data-binary '@filename.json'
and got error
Warning: Couldn't read data from file "filename.json", this makes
Warning: an empty POST.
{
"error" : {
"root_cause" : [ {
"type" : "parse_exception",
"reason" : "Failed to derive xcontent"
} ],
"type" : "parse_exception",
"reason" : "Failed to derive xcontent"
},
"status" : 400
}

i think problem here is in my "text_content" field,because it contains so many type of character .so how i load data in ES?

13

The body part must be on a single line. I think this is your issue here.

1 Like
14

Also you have another problem with CURL. May be running on Windows?
Whatever, have a look at curl documentation to see how to pass a document. Otherwise, switch to SENSE / CONSOLE in Kibana which is easier to use.

1 Like
15

Hi dadoonet,
curl command is running good with other json file...i think problem here is to how to capture the "text_content" field text to form right json format..can you please help me for that?

16

If your JSON document looks exactly like what you pasted, I doubt you can index it for this reason.