The bulk request must be terminated by a newline

[DrJHD]

Despite reading the answers to the many variants of this question, I am stuck. I have tried 1, 2 and 3 "0a" characters at the end of my file (cut down - I have >40 files) and get the same results. I must be missing something obvious, but I can't think what. My terminal session follows.

TIA,

John

dashboard@dashboard:~/es$ hexyl test.json
┌────────┬─────────────────────────┬─────────────────────────┬────────┬────────┐
│00000000│ 7b 20 22 69 6e 64 65 78 ┊ 22 20 3a 20 7b 20 22 5f │{ "index┊" : { "_│
│00000010│ 69 6e 64 65 78 22 20 3a ┊ 20 22 62 6f 6f 6b 73 22 │index" :┊ "books"│
│00000020│ 2c 20 22 5f 69 64 22 20 ┊ 3a 20 22 31 22 20 7d 20 │, "_id" ┊: "1" } │
│00000030│ 7d 0a 7b 22 66 69 6c 65 ┊ 22 3a 22 45 3a 5c 5c 42 │}_{"file┊":"E:\\B│
│00000040│ 4f 4f 4b 53 2f 41 4e 44 ┊ 45 52 53 45 4e 2f 53 41 │OOKS/AND┊ERSEN/SA│
│00000050│ 55 43 59 5f 42 4f 22 2c ┊ 22 63 6f 6e 74 65 6e 74 │UCY_BO",┊"content│
│00000060│ 73 22 3a 22 20 54 48 45 ┊ 20 45 4e 44 5c 6e 22 7d │s":" THE┊ END\n"}│
│00000070│ 0a 0a                   ┊                         │__      ┊        │
└────────┴─────────────────────────┴─────────────────────────┴────────┴────────┘
dashboard@dashboard:~/es$ curl --data-binary -v -T test.json -H 'Content-Type: application/json' http://worker1:31256/books/_bulk?format=yaml
---
error:
  root_cause:
  - type: "illegal_argument_exception"
    reason: "The bulk request must be terminated by a newline [\\n]"
  type: "illegal_argument_exception"
  reason: "The bulk request must be terminated by a newline [\\n]"
status: 400
dashboard@dashboard:~/es$ curl --data-binary -v -T test.json -H 'Content-Type: application/x-ndjson' http://worker1:31256/books/_bulk?format=yaml
---
error:
  root_cause:
  - type: "illegal_argument_exception"
    reason: "The bulk request must be terminated by a newline [\\n]"
  type: "illegal_argument_exception"
  reason: "The bulk request must be terminated by a newline [\\n]"
status: 400
dashboard@dashboard:~/es$ curl http://worker1:31256/_cat/indices
yellow open books HgdlldNLSPG_XyfNyYrWqA 1 1 0 0 208b 208b
dashboard@dashboard:~/es$ curl http://worker1:31256/
{
  "name" : "elasticsearch-7c4b4f7bd9-2crz8",
  "cluster_name" : "docker-cluster",
  "cluster_uuid" : "sLqJwAqLShmZawWMaI-KXg",
  "version" : {
    "number" : "7.13.4",
    "build_flavor" : "default",
    "build_type" : "docker",
    "build_hash" : "c5f60e894ca0c61cdbae4f5a686d9f08bcefc942",
    "build_date" : "2021-07-14T18:33:36.673943207Z",
    "build_snapshot" : false,
    "lucene_version" : "8.8.2",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

Update to clarify the file contents:

dashboard@dashboard:~/es$ cat test.json 
{ "index" : { "_index" : "books", "_id" : "1" } }
{"file":"E:\\BOOKS/ANDERSEN/SAUCY_BO","contents":" THE END\n"}

dashboard@dashboard:~/es$

[spinscale (Alexander Reelsen)]

unless I am misinterpreting that asciidump, you are ending your JSON data value with a newline, but you need to add a newline after all your json data... so " THE┊ END"}\n instead of " THE┊ END\n"}

[DrJHD]

I think one of us is misinterpreting. The "\n" inside the JSON is 5c 6e, i.e. an escaped sequence. The two 0a characters (in this version) are after the closing quote and brace. They appear as underscores in the right hand side of the dump.

[DavidTurner]

Your curl command line is broken, it's setting Content-Length: 2 so Elasticsearch is only reading the first two bytes of the request. I think the combination --data-binary -v means it's trying to send the literal string -v, which the -T is then overriding. Try this instead:

curl -v 'http://worker1:31256/books/_bulk?format=yaml' -H 'Content-type: application/x-ndjson' --data-binary @test.json

[DrJHD]

Thank you. That was the problem. You have saved me hours.

Regards,

John