How to insert data from a json file using bulk API in ES2.1


(Vikas Gopal) #1

Hi Experts,

I have downloaded accounts.json from here and saved it in my local drive . Path is E:\data\accounts.json

Now when I fired
POST /bank/account/_bulk?pretty --data-binary E:\data\accounts.json
I am getting below error
{
"error": {
"root_cause": [
{
"type": "parse_exception",
"reason": "Failed to derive xcontent"
}
],
"type": "parse_exception",
"reason": "Failed to derive xcontent"
},
"status": 400
}

Please suggest what needs to be done here . I believe bulk command is not able to pick the file path.


(David Pilato) #2

accounts ? shakespeare ?

I'm lost


(Vikas Gopal) #3

oops my mistake actually i was trying to insert data for both ,that is account and shakespeare.

For now I want to learn it for accounts , I download it from here and saved it my local drive i.e E:/data/accounts.json.

My question is how I tell bulk API that the above json file is at what path ?


#4

i use this code

$j = array(
"association_key" => $k['id'],
"text" => $k['text'],
"photo" => $k['photo_tweet']);
$j = json_encode($j);
$json_doc.='
{ "create" : { "_index" : "'.$index.'", "_type" : "'.$doc_type.'", "_id" : "0" }}
'.$j.'';
$baseUri = 'http://' . $search_host . ':' . $search_port . '/_bulk';
$ci = curl_init();
curl_setopt($ci, CURLOPT_URL, $baseUri);
curl_setopt($ci, CURLOPT_PORT, $search_port);
curl_setopt($ci, CURLOPT_TIMEOUT, 200);
curl_setopt($ci, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ci, CURLOPT_FORBID_REUSE, 0);
curl_setopt($ci, CURLOPT_CUSTOMREQUEST, 'PUT');
curl_setopt($ci, CURLOPT_POSTFIELDS, $json_doc);
curl_exec($ci);
curl_close($ci);


(Vikas Gopal) #5

Thanks Ryodo,

But how to run this code ? Can't we just do directly the way it explained in the ES Documentation
https://www.elastic.co/guide/en/kibana/current/getting-started.html.

I am just struggling to tell ES the path of json file


(Vikas Gopal) #6

@David

I Edited my question .


(David Pilato) #7

Are you doing that from SENSE? If so it won't work.


(Vikas Gopal) #8

yes I am doing it from SENSE , is there any other way to do this ?


(David Pilato) #9

You can not use curl specific parameters in SENSE.

You can copy and paste the lines of your bulk file just after the _bulk command.
Or you can use curl on the command line.


(Mag Carl) #10

HI Vikas
Can you please explain me how you got around the issue. Im doing same tutotrial..shakespear...and im am getting same error with Sense


(Vikas Gopal) #11

@Mag_Carl, So as David suggested I used Curl rather Sense to fire bulk API.


(Pritam Singh) #12

{"index":{"_id":"58242012e1bc2218ea17f19d"}}
{"Vertical_id":32,
"Vertical_Name":"Ad Servers","Tech_ID":6603,
"Title":"Smart AdServer (Ad Servers)",
"Technology_Name":"Smart AdServer",
"Patterns":{
"Type":"File","Pattern":"cdn1.smartadserver.com/diff,smartadserver.com/call\n",
"text_content":"http://cdn1.smartadserver.com/diff/js/smartadserver.js"\u003e\u003c/script\u003e /javascript", src="http://ww619.smartadserver.com/call/pubj/32872/(home)/13955/m/799""}
}}

i have this type of document, and i am using bulk api like this
curl -XPOST 'localhost:9200/test/test_tech/_bulk?pretty' --data-binary '@filename.json'
and got error
Warning: Couldn't read data from file "filename.json", this makes
Warning: an empty POST.
{
"error" : {
"root_cause" : [ {
"type" : "parse_exception",
"reason" : "Failed to derive xcontent"
} ],
"type" : "parse_exception",
"reason" : "Failed to derive xcontent"
},
"status" : 400
}


i think problem here is in my "text_content" field,because it contains so many type of character .so how i load data in ES?


(David Pilato) #13

The body part must be on a single line. I think this is your issue here.


(David Pilato) #14

Also you have another problem with CURL. May be running on Windows?
Whatever, have a look at curl documentation to see how to pass a document. Otherwise, switch to SENSE / CONSOLE in Kibana which is easier to use.


(Pritam Singh) #15

Hi dadoonet,
curl command is running good with other json file...i think problem here is to how to capture the "text_content" field text to form right json format..can you please help me for that?


(David Pilato) #16

If your JSON document looks exactly like what you pasted, I doubt you can index it for this reason.


(system) #17