How to insert data from a json file using bulk API in ES2.1

(Vikas Gopal) #1

Hi Experts,

I have downloaded accounts.json from here and saved it in my local drive . Path is E:\data\accounts.json

Now when I fired
POST /bank/account/_bulk?pretty --data-binary E:\data\accounts.json
I am getting below error
"error": {
"root_cause": [
"type": "parse_exception",
"reason": "Failed to derive xcontent"
"type": "parse_exception",
"reason": "Failed to derive xcontent"
"status": 400

Please suggest what needs to be done here . I believe bulk command is not able to pick the file path.

(David Pilato) #2

accounts ? shakespeare ?

I'm lost

(Vikas Gopal) #3

oops my mistake actually i was trying to insert data for both ,that is account and shakespeare.

For now I want to learn it for accounts , I download it from here and saved it my local drive i.e E:/data/accounts.json.

My question is how I tell bulk API that the above json file is at what path ?


i use this code

$j = array(
"association_key" => $k['id'],
"text" => $k['text'],
"photo" => $k['photo_tweet']);
$j = json_encode($j);
{ "create" : { "_index" : "'.$index.'", "_type" : "'.$doc_type.'", "_id" : "0" }}
$baseUri = 'http://' . $search_host . ':' . $search_port . '/_bulk';
$ci = curl_init();
curl_setopt($ci, CURLOPT_URL, $baseUri);
curl_setopt($ci, CURLOPT_PORT, $search_port);
curl_setopt($ci, CURLOPT_TIMEOUT, 200);
curl_setopt($ci, CURLOPT_RETURNTRANSFER, 1);
curl_setopt($ci, CURLOPT_FORBID_REUSE, 0);
curl_setopt($ci, CURLOPT_CUSTOMREQUEST, 'PUT');
curl_setopt($ci, CURLOPT_POSTFIELDS, $json_doc);

(Vikas Gopal) #5

Thanks Ryodo,

But how to run this code ? Can't we just do directly the way it explained in the ES Documentation

I am just struggling to tell ES the path of json file

(Vikas Gopal) #6


I Edited my question .

(David Pilato) #7

Are you doing that from SENSE? If so it won't work.

(Vikas Gopal) #8

yes I am doing it from SENSE , is there any other way to do this ?

(David Pilato) #9

You can not use curl specific parameters in SENSE.

You can copy and paste the lines of your bulk file just after the _bulk command.
Or you can use curl on the command line.

(Mag Carl) #10

HI Vikas
Can you please explain me how you got around the issue. Im doing same tutotrial..shakespear...and im am getting same error with Sense

(Vikas Gopal) #11

@Mag_Carl, So as David suggested I used Curl rather Sense to fire bulk API.

(Pritam Singh) #12

"Vertical_Name":"Ad Servers","Tech_ID":6603,
"Title":"Smart AdServer (Ad Servers)",
"Technology_Name":"Smart AdServer",
"text_content":""\u003e\u003c/script\u003e /javascript", src="""}

i have this type of document, and i am using bulk api like this
curl -XPOST 'localhost:9200/test/test_tech/_bulk?pretty' --data-binary '@filename.json'
and got error
Warning: Couldn't read data from file "filename.json", this makes
Warning: an empty POST.
"error" : {
"root_cause" : [ {
"type" : "parse_exception",
"reason" : "Failed to derive xcontent"
} ],
"type" : "parse_exception",
"reason" : "Failed to derive xcontent"
"status" : 400

i think problem here is in my "text_content" field,because it contains so many type of character .so how i load data in ES?

(David Pilato) #13

The body part must be on a single line. I think this is your issue here.

(David Pilato) #14

Also you have another problem with CURL. May be running on Windows?
Whatever, have a look at curl documentation to see how to pass a document. Otherwise, switch to SENSE / CONSOLE in Kibana which is easier to use.

(Pritam Singh) #15

Hi dadoonet,
curl command is running good with other json file...i think problem here is to how to capture the "text_content" field text to form right json format..can you please help me for that?

(David Pilato) #16

If your JSON document looks exactly like what you pasted, I doubt you can index it for this reason.

(system) #17