Bulk via curl not working in data in file?

Elastic Stack Elasticsearch
1

Hi,

I am having troubles executing bulk operations via curl if the bulk command
sequence is in file.

The following works fine:

curl -XDELETE 'http://localhost:9200/test/?pretty=1'

curl -XPUT 'http://localhost:9200/test/?pretty=1' -d '{ index : {
number_of_shards : 3, number_of_replicas : 0 }}'

curl -XPUT 'http://localhost:9200/_bulk' -d '
{ "create" : {"_index":"test","_type":"one","_id":"1"} }
{"name":"01","category":"01","subcategory":"01" }
{ "create" : {"_index":"test","_type":"one","_id":"2"} }
{"name":"02","category":"01","subcategory":"02" }
{ "create" : {"_index":"test","_type":"one","_id":"3"} }
{"name":"03","category":"01","subcategory":"02" }
{ "create" : {"_index":"test","_type":"one","_id":"4"} }
{"name":"04","category":"02","subcategory":"01" }
{ "create" : {"_index":"test","_type":"one","_id":"5"} }
{"name":"05","category":"02","subcategory":"01" }
{ "create" : {"_index":"test","_type":"one","_id":"6"} }
{"name":"06","category":"01","subcategory":"02" }

'

gives me the following response:
{
"ok" : true,
"acknowledged" : true
}
{
"ok" : true,
"acknowledged" : true
}
{"items":[{"create":{"_index":"test","_type":"one","_id":"1","ok":true}},{"create":{"_index":"test","_type":"one","_id":"2","ok":true}},{"create":{"_index":"test","_type":"one","_id":"3","ok":true}},{"create":{"_index":"test","_type":"one","_id":"4","ok":true}},{"create":{"_index":"test","_type":"one","_id":"5","ok":true}},{"create":{"_index":"test","_type":"one","_id":"6","ok":true}}]}

but if I put bulk operations into a file and run

curl -XPUT 'http://localhost:9200/_bulk' -d @bulk.data

then it gives me:

{"error":"ActionRequestValidationException[Validation Failed: 1: no requests
added]"}

Content of the file is the following:

$ cat bulk.data
{ "create" : {"_index":"test","_type":"one","_id":"1"} }
{"name":"01","category":"01","subcategory":"01" }
{ "create" : {"_index":"test","_type":"one","_id":"2"} }
{"name":"02","category":"01","subcategory":"02" }
{ "create" : {"_index":"test","_type":"one","_id":"3"} }
{"name":"03","category":"01","subcategory":"02" }
{ "create" : {"_index":"test","_type":"one","_id":"4"} }
{"name":"04","category":"02","subcategory":"01" }
{ "create" : {"_index":"test","_type":"one","_id":"5"} }
{"name":"05","category":"02","subcategory":"01" }
{ "create" : {"_index":"test","_type":"one","_id":"6"} }
{"name":"06","category":"01","subcategory":"02" }

$

(note the extra new line at the end of the file data)

Regards,
Lukas

2

Ah, the subject should have been: "Bulk via curl not working IF data in
file?"

On Fri, Oct 29, 2010 at 5:32 PM, Lukáš Vlček lukas.vlcek@gmail.com wrote:

Hi,

I am having troubles executing bulk operations via curl if the bulk command
sequence is in file.

The following works fine:

curl -XDELETE 'http://localhost:9200/test/?pretty=1'

curl -XPUT 'http://localhost:9200/test/?pretty=1' -d '{ index : {
number_of_shards : 3, number_of_replicas : 0 }}'

curl -XPUT 'http://localhost:9200/_bulk' -d '
{ "create" : {"_index":"test","_type":"one","_id":"1"} }
{"name":"01","category":"01","subcategory":"01" }
{ "create" : {"_index":"test","_type":"one","_id":"2"} }
{"name":"02","category":"01","subcategory":"02" }
{ "create" : {"_index":"test","_type":"one","_id":"3"} }
{"name":"03","category":"01","subcategory":"02" }
{ "create" : {"_index":"test","_type":"one","_id":"4"} }
{"name":"04","category":"02","subcategory":"01" }
{ "create" : {"_index":"test","_type":"one","_id":"5"} }
{"name":"05","category":"02","subcategory":"01" }
{ "create" : {"_index":"test","_type":"one","_id":"6"} }
{"name":"06","category":"01","subcategory":"02" }

'

gives me the following response:
{
"ok" : true,
"acknowledged" : true
}
{
"ok" : true,
"acknowledged" : true
}

{"items":[{"create":{"_index":"test","_type":"one","_id":"1","ok":true}},{"create":{"_index":"test","_type":"one","_id":"2","ok":true}},{"create":{"_index":"test","_type":"one","_id":"3","ok":true}},{"create":{"_index":"test","_type":"one","_id":"4","ok":true}},{"create":{"_index":"test","_type":"one","_id":"5","ok":true}},{"create":{"_index":"test","_type":"one","_id":"6","ok":true}}]}

but if I put bulk operations into a file and run

curl -XPUT 'http://localhost:9200/_bulk' -d @bulk.data

then it gives me:

{"error":"ActionRequestValidationException[Validation Failed: 1: no
requests added]"}

Content of the file is the following:

$ cat bulk.data
{ "create" : {"_index":"test","_type":"one","_id":"1"} }
{"name":"01","category":"01","subcategory":"01" }
{ "create" : {"_index":"test","_type":"one","_id":"2"} }
{"name":"02","category":"01","subcategory":"02" }
{ "create" : {"_index":"test","_type":"one","_id":"3"} }
{"name":"03","category":"01","subcategory":"02" }
{ "create" : {"_index":"test","_type":"one","_id":"4"} }
{"name":"04","category":"02","subcategory":"01" }
{ "create" : {"_index":"test","_type":"one","_id":"5"} }
{"name":"05","category":"02","subcategory":"01" }
{ "create" : {"_index":"test","_type":"one","_id":"6"} }
{"name":"06","category":"01","subcategory":"02" }

$

(note the extra new line at the end of the file data)

Regards,
Lukas

3

I am running Linux (fedora)

On Fri, Oct 29, 2010 at 5:36 PM, Lukáš Vlček lukas.vlcek@gmail.com wrote:

Ah, the subject should have been: "Bulk via curl not working IF data in
file?"

On Fri, Oct 29, 2010 at 5:32 PM, Lukáš Vlček lukas.vlcek@gmail.comwrote:

Hi,

I am having troubles executing bulk operations via curl if the bulk
command sequence is in file.

The following works fine:

curl -XDELETE 'http://localhost:9200/test/?pretty=1'

curl -XPUT 'http://localhost:9200/test/?pretty=1' -d '{ index : {
number_of_shards : 3, number_of_replicas : 0 }}'

curl -XPUT 'http://localhost:9200/_bulk' -d '
{ "create" : {"_index":"test","_type":"one","_id":"1"} }
{"name":"01","category":"01","subcategory":"01" }
{ "create" : {"_index":"test","_type":"one","_id":"2"} }
{"name":"02","category":"01","subcategory":"02" }
{ "create" : {"_index":"test","_type":"one","_id":"3"} }
{"name":"03","category":"01","subcategory":"02" }
{ "create" : {"_index":"test","_type":"one","_id":"4"} }
{"name":"04","category":"02","subcategory":"01" }
{ "create" : {"_index":"test","_type":"one","_id":"5"} }
{"name":"05","category":"02","subcategory":"01" }
{ "create" : {"_index":"test","_type":"one","_id":"6"} }
{"name":"06","category":"01","subcategory":"02" }

'

gives me the following response:
{
"ok" : true,
"acknowledged" : true
}
{
"ok" : true,
"acknowledged" : true
}

{"items":[{"create":{"_index":"test","_type":"one","_id":"1","ok":true}},{"create":{"_index":"test","_type":"one","_id":"2","ok":true}},{"create":{"_index":"test","_type":"one","_id":"3","ok":true}},{"create":{"_index":"test","_type":"one","_id":"4","ok":true}},{"create":{"_index":"test","_type":"one","_id":"5","ok":true}},{"create":{"_index":"test","_type":"one","_id":"6","ok":true}}]}

but if I put bulk operations into a file and run

curl -XPUT 'http://localhost:9200/_bulk' -d @bulk.data

then it gives me:

{"error":"ActionRequestValidationException[Validation Failed: 1: no
requests added]"}

Content of the file is the following:

$ cat bulk.data
{ "create" : {"_index":"test","_type":"one","_id":"1"} }
{"name":"01","category":"01","subcategory":"01" }
{ "create" : {"_index":"test","_type":"one","_id":"2"} }
{"name":"02","category":"01","subcategory":"02" }
{ "create" : {"_index":"test","_type":"one","_id":"3"} }
{"name":"03","category":"01","subcategory":"02" }
{ "create" : {"_index":"test","_type":"one","_id":"4"} }
{"name":"04","category":"02","subcategory":"01" }
{ "create" : {"_index":"test","_type":"one","_id":"5"} }
{"name":"05","category":"02","subcategory":"01" }
{ "create" : {"_index":"test","_type":"one","_id":"6"} }
{"name":"06","category":"01","subcategory":"02" }

$

(note the extra new line at the end of the file data)

Regards,
Lukas

4

So the trick is to use --data-binary
curl -XPUT 'http://localhost:9200/_bulk' --data-binary @bulk.data

thanks @clintongormley

On Fri, Oct 29, 2010 at 5:42 PM, Lukáš Vlček lukas.vlcek@gmail.com wrote:

I am running Linux (fedora)

On Fri, Oct 29, 2010 at 5:36 PM, Lukáš Vlček lukas.vlcek@gmail.comwrote:

Ah, the subject should have been: "Bulk via curl not working IF data in
file?"

On Fri, Oct 29, 2010 at 5:32 PM, Lukáš Vlček lukas.vlcek@gmail.comwrote:

Hi,

I am having troubles executing bulk operations via curl if the bulk
command sequence is in file.

The following works fine:

curl -XDELETE 'http://localhost:9200/test/?pretty=1'

curl -XPUT 'http://localhost:9200/test/?pretty=1' -d '{ index : {
number_of_shards : 3, number_of_replicas : 0 }}'

curl -XPUT 'http://localhost:9200/_bulk' -d '
{ "create" : {"_index":"test","_type":"one","_id":"1"} }
{"name":"01","category":"01","subcategory":"01" }
{ "create" : {"_index":"test","_type":"one","_id":"2"} }
{"name":"02","category":"01","subcategory":"02" }
{ "create" : {"_index":"test","_type":"one","_id":"3"} }
{"name":"03","category":"01","subcategory":"02" }
{ "create" : {"_index":"test","_type":"one","_id":"4"} }
{"name":"04","category":"02","subcategory":"01" }
{ "create" : {"_index":"test","_type":"one","_id":"5"} }
{"name":"05","category":"02","subcategory":"01" }
{ "create" : {"_index":"test","_type":"one","_id":"6"} }
{"name":"06","category":"01","subcategory":"02" }

'

gives me the following response:
{
"ok" : true,
"acknowledged" : true
}
{
"ok" : true,
"acknowledged" : true
}

{"items":[{"create":{"_index":"test","_type":"one","_id":"1","ok":true}},{"create":{"_index":"test","_type":"one","_id":"2","ok":true}},{"create":{"_index":"test","_type":"one","_id":"3","ok":true}},{"create":{"_index":"test","_type":"one","_id":"4","ok":true}},{"create":{"_index":"test","_type":"one","_id":"5","ok":true}},{"create":{"_index":"test","_type":"one","_id":"6","ok":true}}]}

but if I put bulk operations into a file and run

curl -XPUT 'http://localhost:9200/_bulk' -d @bulk.data

then it gives me:

{"error":"ActionRequestValidationException[Validation Failed: 1: no
requests added]"}

Content of the file is the following:

$ cat bulk.data
{ "create" : {"_index":"test","_type":"one","_id":"1"} }
{"name":"01","category":"01","subcategory":"01" }
{ "create" : {"_index":"test","_type":"one","_id":"2"} }
{"name":"02","category":"01","subcategory":"02" }
{ "create" : {"_index":"test","_type":"one","_id":"3"} }
{"name":"03","category":"01","subcategory":"02" }
{ "create" : {"_index":"test","_type":"one","_id":"4"} }
{"name":"04","category":"02","subcategory":"01" }
{ "create" : {"_index":"test","_type":"one","_id":"5"} }
{"name":"05","category":"02","subcategory":"01" }
{ "create" : {"_index":"test","_type":"one","_id":"6"} }
{"name":"06","category":"01","subcategory":"02" }

$

(note the extra new line at the end of the file data)

Regards,
Lukas

5

thanks for posting the solution! I was having the same issue.

On Fri, Oct 29, 2010 at 10:58 AM, Lukáš Vlček lukas.vlcek@gmail.com wrote:

So the trick is to use --data-binary
curl -XPUT 'http://localhost:9200/_bulk' --data-binary @bulk.data

thanks @clintongormley

On Fri, Oct 29, 2010 at 5:42 PM, Lukáš Vlček lukas.vlcek@gmail.comwrote:

I am running Linux (fedora)

On Fri, Oct 29, 2010 at 5:36 PM, Lukáš Vlček lukas.vlcek@gmail.comwrote:

Ah, the subject should have been: "Bulk via curl not working IF data in
file?"

On Fri, Oct 29, 2010 at 5:32 PM, Lukáš Vlček lukas.vlcek@gmail.comwrote:

Hi,

I am having troubles executing bulk operations via curl if the bulk
command sequence is in file.

The following works fine:

curl -XDELETE 'http://localhost:9200/test/?pretty=1'

curl -XPUT 'http://localhost:9200/test/?pretty=1' -d '{ index : {
number_of_shards : 3, number_of_replicas : 0 }}'

curl -XPUT 'http://localhost:9200/_bulk' -d '
{ "create" : {"_index":"test","_type":"one","_id":"1"} }
{"name":"01","category":"01","subcategory":"01" }
{ "create" : {"_index":"test","_type":"one","_id":"2"} }
{"name":"02","category":"01","subcategory":"02" }
{ "create" : {"_index":"test","_type":"one","_id":"3"} }
{"name":"03","category":"01","subcategory":"02" }
{ "create" : {"_index":"test","_type":"one","_id":"4"} }
{"name":"04","category":"02","subcategory":"01" }
{ "create" : {"_index":"test","_type":"one","_id":"5"} }
{"name":"05","category":"02","subcategory":"01" }
{ "create" : {"_index":"test","_type":"one","_id":"6"} }
{"name":"06","category":"01","subcategory":"02" }

'

gives me the following response:
{
"ok" : true,
"acknowledged" : true
}
{
"ok" : true,
"acknowledged" : true
}

{"items":[{"create":{"_index":"test","_type":"one","_id":"1","ok":true}},{"create":{"_index":"test","_type":"one","_id":"2","ok":true}},{"create":{"_index":"test","_type":"one","_id":"3","ok":true}},{"create":{"_index":"test","_type":"one","_id":"4","ok":true}},{"create":{"_index":"test","_type":"one","_id":"5","ok":true}},{"create":{"_index":"test","_type":"one","_id":"6","ok":true}}]}

but if I put bulk operations into a file and run

curl -XPUT 'http://localhost:9200/_bulk' -d @bulk.data

then it gives me:

{"error":"ActionRequestValidationException[Validation Failed: 1: no
requests added]"}

Content of the file is the following:

$ cat bulk.data
{ "create" : {"_index":"test","_type":"one","_id":"1"} }
{"name":"01","category":"01","subcategory":"01" }
{ "create" : {"_index":"test","_type":"one","_id":"2"} }
{"name":"02","category":"01","subcategory":"02" }
{ "create" : {"_index":"test","_type":"one","_id":"3"} }
{"name":"03","category":"01","subcategory":"02" }
{ "create" : {"_index":"test","_type":"one","_id":"4"} }
{"name":"04","category":"02","subcategory":"01" }
{ "create" : {"_index":"test","_type":"one","_id":"5"} }
{"name":"05","category":"02","subcategory":"01" }
{ "create" : {"_index":"test","_type":"one","_id":"6"} }
{"name":"06","category":"01","subcategory":"02" }

$

(note the extra new line at the end of the file data)

Regards,
Lukas

6

I would note that many articles out there seem to indicate you need to urlencode the contents of a datafile being posted via curl in binary mode.

As you have found -- and I have found -- this is not necessary when posting JSON as content.

7

I needed to do something similar from within Ruby code w/o writing to a file first, so see code below for an example:

example json data generated by the code below, please note that the meta fields and source fields are separated by a newline (http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/docs-bulk.html)

=begin
{:index => {"_index" => "xxx", "_type" => "yyy", "_id" => "xxx-yyy-zzz"}}
{:field0 => [{"field1" => "value1", "field2" => "value2", "field3" => "value3"}, {"field1" => "value1", "field2" => "value2", "field3" => "value3"}, {"field1" => "value1", "field2" => "value2", "field3" => "value3"}]}
=end

host = 'localhost'

bulk_contents << {:index => meta_data_fields} # meta data fields contains fields such as _id, _type, and _index

documents << document_fields # fields specific to the document to be modified

bulk_contents << {:field0 => documents} # please see above of an example of the generated json. The 'field0' key is part of a partial document

bulk_contents_as_string = bulk_contents.map(&:to_json).join("\n")

bulk_update_request = "curl -s -XPOST 'http://#{host}:9200/_bulk' --data-binary '#{bulk_contents_as_string}\n'" # terminated by a newline

system("#{bulk_update_request} > /dev/null") # or system(bulk_update_request) if you require the output