Error when loading the json file with the curl command

Elastic Stack Elasticsearch
1

Hi, I have to analyze some packages captured with wireshark on kibana. To do this I am following the guide:
https://www.elastic.co/blog/analyzing-network-packets-with-wireshark-elasticsearch-and-kibana

The problem occurs when I import the .json file to elasticsearch with the curl command. The command which I launch is:
curl -vv -H "Content-Type: application/json" -XPOST "localhost:9200/_bulk" --data-binary "@C:\Users\Thebe\Desktop\singolopacchetto.json"

The answer which I receive is:
{"error":{"root_cause":[{"type":"json_e_o_f_exception","reason":"Unexpected end-of-input: expected close marker for Object (start marker at [Source: org.elasticsearch.transport.netty4.ByteBufStreamInput@74dec268; line: 1, column: 1])\n at [Source: org.elasticsearch.transport.netty4.ByteBufStreamInput@74dec268; line: 2, column: 3]"}],"type":"json_e_o_f_exception","reason":"Unexpected end-of-input: expected close marker for Object (start marker at [Source: org.elasticsearch.transport.netty4.ByteBufStreamInput@74dec268; line: 1, column: 1])\n at [Source: org.elasticsearch.transport.netty4.ByteBufStreamInput@74dec268; line: 2, column: 3]"},"status":500}* Connection #0 to host localhost left intact

I loaded the json file on drive: https://drive.google.com/file/d/1XH8miHBjp47a5HCA1_iE7K_VY8fsK9QC/view?usp=sharing

The version of elasticsearch and kibana I'm using is 5.6.9
What is the problem? And how can I solve?

2

The Bulk API expects newline-delimited JSON of the following form:

action_and_meta_data\n
optional_source\n
action_and_meta_data\n
optional_source\n
....
action_and_meta_data\n
optional_source\n

Either index this document as a single document, or else format it correctly for bulk processing:

{"index":{"_index":"packets-2018-04-25","_type":"pcap_file"}}
{"layers":{"frame":{"frame.interface_id":"0","frame.interface_id_tree":{"frame.interface_name":"any"},"frame.encap_type":"25","frame.time":"Apr 20, 2018 15:30:52.669797277 ora legale Europa occidentale","frame.offset_shift":"0.000000000","frame.time_epoch":"1524231052.669797277","frame.time_delta":"0.000000000","frame.time_delta_displayed":"0.000000000","frame.time_relative":"0.000000000","frame.number":"1","frame.len":"649","frame.cap_len":"649","frame.marked":"0","frame.ignored":"0","frame.protocols":"sll:ethertype:ip:tcp:http:json","frame.coloring_rule.name":"HTTP","frame.coloring_rule.string":"http || tcp.port == 80 || http2"},"sll":{"sll.pkttype":"0","sll.hatype":"772","sll.halen":"6","sll.src.eth":"00:00:00:00:00:00","sll.unused":"00:00","sll.etype":"0x00000800"},"ip":{"ip.version":"4","ip.hdr_len":"20","ip.dsfield":"0x00000000","ip.dsfield_tree":{"ip.dsfield.dscp":"0","ip.dsfield.ecn":"0"},"ip.len":"633","ip.id":"0x0000b60a","ip.flags":"0x00000002","ip.flags_tree":{"ip.flags.rb":"0","ip.flags.df":"1","ip.flags.mf":"0"},"ip.frag_offset":"0","ip.ttl":"64","ip.proto":"6","ip.checksum":"0x00008472","ip.checksum.status":"2","ip.src":"127.0.0.1","ip.addr":"127.0.0.1","ip.src_host":"127.0.0.1","ip.host":"127.0.0.1","ip.dst":"127.0.0.1","ip.dst_host":"127.0.0.1","Source GeoIP: Unknown":"","Destination GeoIP: Unknown":""},"tcp":{"tcp.srcport":"35474","tcp.dstport":"9200","tcp.port":"9200","tcp.stream":"0","tcp.len":"581","tcp.seq":"1","tcp.nxtseq":"582","tcp.ack":"1","tcp.hdr_len":"32","tcp.flags":"0x00000018","tcp.flags_tree":{"tcp.flags.res":"0","tcp.flags.ns":"0","tcp.flags.cwr":"0","tcp.flags.ecn":"0","tcp.flags.urg":"0","tcp.flags.ack":"1","tcp.flags.push":"1","tcp.flags.reset":"0","tcp.flags.syn":"0","tcp.flags.fin":"0","tcp.flags.str":"·······AP···"},"tcp.window_size_value":"3637","tcp.window_size":"3637","tcp.window_size_scalefactor":"-1","tcp.checksum":"0x0000006e","tcp.checksum.status":"2","tcp.urgent_pointer":"0","tcp.options":"01:01:08:0a:ca:ca:b1:90:ca:ca:af:11","tcp.options_tree":{"tcp.options.nop":"01","tcp.options.nop_tree":{"tcp.option_kind":"1"},"tcp.options.timestamp":"08:0a:ca:ca:b1:90:ca:ca:af:11","tcp.options.timestamp_tree":{"tcp.option_kind":"8","tcp.option_len":"10","tcp.options.timestamp.tsval":"3402281360","tcp.options.timestamp.tsecr":"3402280721"}},"tcp.analysis":{"tcp.analysis.bytes_in_flight":"581","tcp.analysis.push_bytes_sent":"581"},"tcp.payload":"50:4f:53:54:20:2f:2e:72:65:70:6f:72:74:69:6e:67:2d:2a:2f:65:73:71:75:65:75:65:2f:5f:73:65:61:72:63:68:3f:76:65:72:73:69:6f:6e:3d:74:72:75:65:20:48:54:54:50:2f:31:2e:31:0d:0a:41:75:74:68:6f:72:69:7a:61:74:69:6f:6e:3a:20:42:61:73:69:63:20:61:32:6c:69:59:57:35:68:4f:6d:74:70:59:6d:46:75:59:58:42:68:63:33:4e:33:62:33:4a:6b:0d:0a:63:6f:6e:74:65:6e:74:2d:74:79:70:65:3a:20:61:70:70:6c:69:63:61:74:69:6f:6e:2f:6a:73:6f:6e:0d:0a:48:6f:73:74:3a:20:6c:6f:63:61:6c:68:6f:73:74:3a:39:32:30:30:0d:0a:43:6f:6e:74:65:6e:74:2d:4c:65:6e:67:74:68:3a:20:33:37:31:0d:0a:43:6f:6e:6e:65:63:74:69:6f:6e:3a:20:6b:65:65:70:2d:61:6c:69:76:65:0d:0a:0d:0a:7b:22:5f:73:6f:75:72:63:65:22:3a:7b:22:65:78:63:6c:75:64:65:73:22:3a:5b:22:6f:75:74:70:75:74:2e:63:6f:6e:74:65:6e:74:22:5d:7d:2c:22:71:75:65:72:79:22:3a:7b:22:63:6f:6e:73:74:61:6e:74:5f:73:63:6f:72:65:22:3a:7b:22:66:69:6c:74:65:72:22:3a:7b:22:62:6f:6f:6c:22:3a:7b:22:66:69:6c:74:65:72:22:3a:7b:22:74:65:72:6d:22:3a:7b:22:6a:6f:62:74:79:70:65:22:3a:22:63:73:76:22:7d:7d:2c:22:73:68:6f:75:6c:64:22:3a:5b:7b:22:74:65:72:6d:22:3a:7b:22:73:74:61:74:75:73:22:3a:22:70:65:6e:64:69:6e:67:22:7d:7d:2c:7b:22:62:6f:6f:6c:22:3a:7b:22:66:69:6c:74:65:72:22:3a:5b:7b:22:74:65:72:6d:22:3a:7b:22:73:74:61:74:75:73:22:3a:22:70:72:6f:63:65:73:73:69:6e:67:22:7d:7d:2c:7b:22:72:61:6e:67:65:22:3a:7b:22:70:72:6f:63:65:73:73:5f:65:78:70:69:72:61:74:69:6f:6e:22:3a:7b:22:6c:74:65:22:3a:22:32:30:31:38:2d:30:34:2d:32:30:54:31:33:3a:33:30:3a:35:32:2e:36:36:38:5a:22:7d:7d:7d:5d:7d:7d:5d:7d:7d:7d:7d:2c:22:73:6f:72:74:22:3a:5b:7b:22:70:72:69:6f:72:69:74:79:22:3a:7b:22:6f:72:64:65:72:22:3a:22:61:73:63:22:7d:7d:2c:7b:22:63:72:65:61:74:65:64:5f:61:74:22:3a:7b:22:6f:72:64:65:72:22:3a:22:61:73:63:22:7d:7d:5d:2c:22:73:69:7a:65:22:3a:31:30:7d"},"http":{"POST /.reporting-*/esqueue/_search?version=true HTTP/1.1\\r\\n":{"_ws.expert":{"http.chat":"","_ws.expert.message":"POST /.reporting-*/esqueue/_search?version=true HTTP/1.1\\r\\n","_ws.expert.severity":"2097152","_ws.expert.group":"33554432"},"http.request.method":"POST","http.request.uri":"/.reporting-*/esqueue/_search?version=true","http.request.uri_tree":{"http.request.uri.path":"/.reporting-*/esqueue/_search","http.request.uri.query":"version=true","http.request.uri.query_tree":{"http.request.uri.query.parameter":"version=true"}},"http.request.version":"HTTP/1.1"},"http.authorization":"Basic a2liYW5hOmtpYmFuYXBhc3N3b3Jk","http.authorization_tree":{"http.authbasic":"kibana:kibanapassword"},"http.request.line":"Connection: keep-alive\r\n","http.content_type":"application/json","http.host":"localhost:9200","http.content_length_header":"371","http.content_length_header_tree":{"http.content_length":"371"},"http.connection":"keep-alive","\\r\\n":"","http.request.full_uri":"http://localhost:9200/.reporting-*/esqueue/_search?version=true","http.request":"1","http.request_number":"1","http.response_in":"3","http.next_request_in":"14","http.file_data":"{\"_source\":{\"excludes\":[\"output.content\"]},\"query\":{\"constant_score\":{\"filter\":{\"bool\":{\"filter\":{\"term\":{\"jobtype\":\"csv\"}},\"should\":[{\"term\":{\"status\":\"pending\"}},{\"bool\":{\"filter\":[{\"term\":{\"status\":\"processing\"}},{\"range\":{\"process_expiration\":{\"lte\":\"2018-04-20T13:30:52.668Z\"}}}]}}]}}}},\"sort\":[{\"priority\":{\"order\":\"asc\"}},{\"created_at\":{\"order\":\"asc\"}}],\"size\":10}"},"json":{"json.object":{"json.member":{"json.value.number":"10","json.key":"size"}}}}}
3

I created a new json with the format that sent me, but when I load it I receive new error:

{
  "error" : {
    "root_cause" : [
      {
        "type" : "action_request_validation_exception",
        "reason" : "Validation Failed: 1: no requests added;"
      }
    ],
    "type" : "action_request_validation_exception",
    "reason" : "Validation Failed: 1: no requests added;"
  },
  "status" : 400
}
4

Could you send the curl command you're using and the JSON file that you're trying to post? There's something wrong, but without seeing these it's hard to know what.

5

I use this curl command:
curl -H 'Content-Type:application/json' -XPOST "localhost:9200/packets-2018-04-25/pcap_file/_bulk?pretty" --data-binary @C:\Users\Thebe\Desktop\singolopacchetto.json

6

The JSON file would also be useful to see.

7

The json file is what you wrote to me

{"index":{"_index":"packets-2018-04-25","_type":"pcap_file"}}
{"layers":{"frame":{"frame.interface_id":"0","frame.interface_id_tree":{"frame.interface_name":"any"},"frame.encap_type":"25","frame.time":"Apr 20, 2018 15:30:52.669797277 ora legale Europa occidentale","frame.offset_shift":"0.000000000","frame.time_epoch":"1524231052.669797277","frame.time_delta":"0.000000000","frame.time_delta_displayed":"0.000000000","frame.time_relative":"0.000000000","frame.number":"1","frame.len":"649","frame.cap_len":"649","frame.marked":"0","frame.ignored":"0","frame.protocols":"sll:ethertype:ip:tcp:http:json","frame.coloring_rule.name":"HTTP","frame.coloring_rule.string":"http || tcp.port == 80 || http2"},"sll":{"sll.pkttype":"0","sll.hatype":"772","sll.halen":"6","sll.src.eth":"00:00:00:00:00:00","sll.unused":"00:00","sll.etype":"0x00000800"},"ip":{"ip.version":"4","ip.hdr_len":"20","ip.dsfield":"0x00000000","ip.dsfield_tree":{"ip.dsfield.dscp":"0","ip.dsfield.ecn":"0"},"ip.len":"633","ip.id":"0x0000b60a","ip.flags":"0x00000002","ip.flags_tree":{"ip.flags.rb":"0","ip.flags.df":"1","ip.flags.mf":"0"},"ip.frag_offset":"0","ip.ttl":"64","ip.proto":"6","ip.checksum":"0x00008472","ip.checksum.status":"2","ip.src":"127.0.0.1","ip.addr":"127.0.0.1","ip.src_host":"127.0.0.1","ip.host":"127.0.0.1","ip.dst":"127.0.0.1","ip.dst_host":"127.0.0.1","Source GeoIP: Unknown":"","Destination GeoIP: Unknown":""},"tcp":{"tcp.srcport":"35474","tcp.dstport":"9200","tcp.port":"9200","tcp.stream":"0","tcp.len":"581","tcp.seq":"1","tcp.nxtseq":"582","tcp.ack":"1","tcp.hdr_len":"32","tcp.flags":"0x00000018","tcp.flags_tree":{"tcp.flags.res":"0","tcp.flags.ns":"0","tcp.flags.cwr":"0","tcp.flags.ecn":"0","tcp.flags.urg":"0","tcp.flags.ack":"1","tcp.flags.push":"1","tcp.flags.reset":"0","tcp.flags.syn":"0","tcp.flags.fin":"0","tcp.flags.str":"·······AP···"},"tcp.window_size_value":"3637","tcp.window_size":"3637","tcp.window_size_scalefactor":"-1","tcp.checksum":"0x0000006e","tcp.checksum.status":"2","tcp.urgent_pointer":"0","tcp.options":"01:01:08:0a:ca:ca:b1:90:ca:ca:af:11","tcp.options_tree":{"tcp.options.nop":"01","tcp.options.nop_tree":{"tcp.option_kind":"1"},"tcp.options.timestamp":"08:0a:ca:ca:b1:90:ca:ca:af:11","tcp.options.timestamp_tree":{"tcp.option_kind":"8","tcp.option_len":"10","tcp.options.timestamp.tsval":"3402281360","tcp.options.timestamp.tsecr":"3402280721"}},"tcp.analysis":{"tcp.analysis.bytes_in_flight":"581","tcp.analysis.push_bytes_sent":"581"},"tcp.payload":"50:4f:53:54:20:2f:2e:72:65:70:6f:72:74:69:6e:67:2d:2a:2f:65:73:71:75:65:75:65:2f:5f:73:65:61:72:63:68:3f:76:65:72:73:69:6f:6e:3d:74:72:75:65:20:48:54:54:50:2f:31:2e:31:0d:0a:41:75:74:68:6f:72:69:7a:61:74:69:6f:6e:3a:20:42:61:73:69:63:20:61:32:6c:69:59:57:35:68:4f:6d:74:70:59:6d:46:75:59:58:42:68:63:33:4e:33:62:33:4a:6b:0d:0a:63:6f:6e:74:65:6e:74:2d:74:79:70:65:3a:20:61:70:70:6c:69:63:61:74:69:6f:6e:2f:6a:73:6f:6e:0d:0a:48:6f:73:74:3a:20:6c:6f:63:61:6c:68:6f:73:74:3a:39:32:30:30:0d:0a:43:6f:6e:74:65:6e:74:2d:4c:65:6e:67:74:68:3a:20:33:37:31:0d:0a:43:6f:6e:6e:65:63:74:69:6f:6e:3a:20:6b:65:65:70:2d:61:6c:69:76:65:0d:0a:0d:0a:7b:22:5f:73:6f:75:72:63:65:22:3a:7b:22:65:78:63:6c:75:64:65:73:22:3a:5b:22:6f:75:74:70:75:74:2e:63:6f:6e:74:65:6e:74:22:5d:7d:2c:22:71:75:65:72:79:22:3a:7b:22:63:6f:6e:73:74:61:6e:74:5f:73:63:6f:72:65:22:3a:7b:22:66:69:6c:74:65:72:22:3a:7b:22:62:6f:6f:6c:22:3a:7b:22:66:69:6c:74:65:72:22:3a:7b:22:74:65:72:6d:22:3a:7b:22:6a:6f:62:74:79:70:65:22:3a:22:63:73:76:22:7d:7d:2c:22:73:68:6f:75:6c:64:22:3a:5b:7b:22:74:65:72:6d:22:3a:7b:22:73:74:61:74:75:73:22:3a:22:70:65:6e:64:69:6e:67:22:7d:7d:2c:7b:22:62:6f:6f:6c:22:3a:7b:22:66:69:6c:74:65:72:22:3a:5b:7b:22:74:65:72:6d:22:3a:7b:22:73:74:61:74:75:73:22:3a:22:70:72:6f:63:65:73:73:69:6e:67:22:7d:7d:2c:7b:22:72:61:6e:67:65:22:3a:7b:22:70:72:6f:63:65:73:73:5f:65:78:70:69:72:61:74:69:6f:6e:22:3a:7b:22:6c:74:65:22:3a:22:32:30:31:38:2d:30:34:2d:32:30:54:31:33:3a:33:30:3a:35:32:2e:36:36:38:5a:22:7d:7d:7d:5d:7d:7d:5d:7d:7d:7d:7d:2c:22:73:6f:72:74:22:3a:5b:7b:22:70:72:69:6f:72:69:74:79:22:3a:7b:22:6f:72:64:65:72:22:3a:22:61:73:63:22:7d:7d:2c:7b:22:63:72:65:61:74:65:64:5f:61:74:22:3a:7b:22:6f:72:64:65:72:22:3a:22:61:73:63:22:7d:7d:5d:2c:22:73:69:7a:65:22:3a:31:30:7d"},"http":{"POST /.reporting-*/esqueue/_search?version=true HTTP/1.1\\r\\n":{"_ws.expert":{"http.chat":"","_ws.expert.message":"POST /.reporting-*/esqueue/_search?version=true HTTP/1.1\\r\\n","_ws.expert.severity":"2097152","_ws.expert.group":"33554432"},"http.request.method":"POST","http.request.uri":"/.reporting-*/esqueue/_search?version=true","http.request.uri_tree":{"http.request.uri.path":"/.reporting-*/esqueue/_search","http.request.uri.query":"version=true","http.request.uri.query_tree":{"http.request.uri.query.parameter":"version=true"}},"http.request.version":"HTTP/1.1"},"http.authorization":"Basic a2liYW5hOmtpYmFuYXBhc3N3b3Jk","http.authorization_tree":{"http.authbasic":"kibana:kibanapassword"},"http.request.line":"Connection: keep-alive\r\n","http.content_type":"application/json","http.host":"localhost:9200","http.content_length_header":"371","http.content_length_header_tree":{"http.content_length":"371"},"http.connection":"keep-alive","\\r\\n":"","http.request.full_uri":"http://localhost:9200/.reporting-*/esqueue/_search?version=true","http.request":"1","http.request_number":"1","http.response_in":"3","http.next_request_in":"14","http.file_data":"{\"_source\":{\"excludes\":[\"output.content\"]},\"query\":{\"constant_score\":{\"filter\":{\"bool\":{\"filter\":{\"term\":{\"jobtype\":\"csv\"}},\"should\":[{\"term\":{\"status\":\"pending\"}},{\"bool\":{\"filter\":[{\"term\":{\"status\":\"processing\"}},{\"range\":{\"process_expiration\":{\"lte\":\"2018-04-20T13:30:52.668Z\"}}}]}}]}}}},\"sort\":[{\"priority\":{\"order\":\"asc\"}},{\"created_at\":{\"order\":\"asc\"}}],\"size\":10}"},"json":{"json.object":{"json.member":{"json.value.number":"10","json.key":"size"}}}}}
8

I can't reproduce this. Here a copy of a terminal session showing a sequence of curl commands run on a completely fresh Elasticsearch 5.6.8 installation, in which the document is indexed with no errors:

Put template with mapping

$ curl -v -XPUT 'http://localhost:9200/_template/packets' -H 'Content-type: application/json' --data-binary $'{"mappings":{"pcap_file":{"dynamic":"false","properties":{"layers":{"properties":{"ip":{"properties":{"ip_ip_dst":{"type":"ip"},"ip_ip_src":{"type":"ip"}}},"udp":{"properties":{"udp_udp_srcport":{"type":"integer"},"udp_udp_dstport":{"type":"integer"}}},"frame":{"properties":{"frame_frame_protocols":{"type":"keyword"},"frame_frame_len":{"type":"long"}}}}},"timestamp":{"type":"date"}}}},"settings":{"number_of_shards":1,"number_of_replicas":0},"template":"packets-*"}'
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 9200 (#0)
> PUT /_template/packets HTTP/1.1
> Host: localhost:9200
> User-Agent: curl/7.54.0
> Accept: */*
> Content-type: application/json
> Content-Length: 468
> 
* upload completely sent off: 468 out of 468 bytes
< HTTP/1.1 200 OK
< content-type: application/json; charset=UTF-8
< content-length: 21
< 
* Connection #0 to host localhost left intact
{"acknowledged":true}

Show head and tail of file

(sorry, can't show it all, it pushes this message over the size limit)

$ head -c200 singolopacchetto.json 
{"index":{"_index":"packets-2018-04-25","_type":"pcap_file"}}
{"layers":{"frame":{"frame.interface_id":"0","frame.interface_id_tree":{"frame.interface_name":"any"},"frame.encap_type":"25","frame.time"
$ tail -c50 singolopacchetto.json 
:{"json.value.number":"10","json.key":"size"}}}}}

Checksum contents of file

$ shasum singolopacchetto.json
4d603b226395c66b88747f66044b5e47436a2aa9  singolopacchetto.json

Put document using given command line

$ curl -v 'http://localhost:9200/packets-2018-04-25/pcap_file/_bulk?pretty' -H 'Content-type: application/json' --data-binary @singolopacchetto.json
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 9200 (#0)
> POST /packets-2018-04-25/pcap_file/_bulk?pretty HTTP/1.1
> Host: localhost:9200
> User-Agent: curl/7.54.0
> Accept: */*
> Content-type: application/json
> Content-Length: 5957
> Expect: 100-continue
> 
< HTTP/1.1 100 Continue
* We are completely uploaded and fine
< HTTP/1.1 200 OK
< content-type: application/json; charset=UTF-8
< content-length: 418
< 
{
  "took" : 330,
  "errors" : false,
  "items" : [
    {
      "index" : {
        "_index" : "packets-2018-04-25",
        "_type" : "pcap_file",
        "_id" : "AWMXTI7T_QeHdqozGVFr",
        "_version" : 1,
        "result" : "created",
        "_shards" : {
          "total" : 1,
          "successful" : 1,
          "failed" : 0
        },
        "created" : true,
        "status" : 201
      }
    }
  ]
}
* Connection #0 to host localhost left intact

Put document (correct Content-type, and using index & type given in file)

$ curl -v 'http://localhost:9200/_bulk' -H 'Content-type: application/x-ndjson' --data-binary @singolopacchetto.json
*   Trying ::1...
* TCP_NODELAY set
* Connected to localhost (::1) port 9200 (#0)
> POST /_bulk HTTP/1.1
> Host: localhost:9200
> User-Agent: curl/7.54.0
> Accept: */*
> Content-type: application/x-ndjson
> Content-Length: 5957
> Expect: 100-continue
> 
< HTTP/1.1 100 Continue
* We are completely uploaded and fine
< HTTP/1.1 200 OK
< content-type: application/json; charset=UTF-8
< content-length: 235
< 
* Connection #0 to host localhost left intact
{"took":27,"errors":false,"items":[{"index":{"_index":"packets-2018-04-25","_type":"pcap_file","_id":"AWMXTKmh_QeHdqozGVFs","_version":1,"result":"created","_shards":{"total":1,"successful":1,"failed":0},"created":true,"status":201}}]}
9

Apologies, you said you were using 5.6.9. I get the same results on 5.6.9.

Can you share a fuller description of how you're getting the error you quote, similarly to how I've done above? Ideally start with a completely blank Elasticsearch instance and include the commands to create the template and everything else that you do leading up to the message you are getting?

10

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.