"Failed to derive xcontent" on REST bulk with ES 5.0

Elastic Stack Elasticsearch
1

Hi all,
I'm migrating code from 2.4 to 5.0 and I'm facing an issue. The code bellow (that uses RestClient v 5.0.0) fails to execute the whole bulk and raises a

java.lang.reflect.InvocationTargetException
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at org.codehaus.mojo.exec.ExecJavaMojo$1.run(ExecJavaMojo.java:294)
	at java.lang.Thread.run(Thread.java:745)
Caused by: org.elasticsearch.client.ResponseException: POST http://127.0.0.1:9200/beam/test/_bulk?refresh=true: HTTP/1.1 400 Bad Request
{"error":{"root_cause":[{"type":"parse_exception","reason":"Failed to derive xcontent"}],"type":"parse_exception","reason":"Failed to derive xcontent"},"status":400}
	at org.elasticsearch.client.RestClient$1.completed(RestClient.java:310)
	at org.elasticsearch.client.RestClient$1.completed(RestClient.java:299)
	at org.apache.http.concurrent.BasicFuture.completed(BasicFuture.java:119)
	at org.apache.http.impl.nio.client.DefaultClientExchangeHandlerImpl.responseCompleted(DefaultClientExchangeHandlerImpl.java:177)
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.processResponse(HttpAsyncRequestExecutor.java:436)
	at org.apache.http.nio.protocol.HttpAsyncRequestExecutor.inputReady(HttpAsyncRequestExecutor.java:326)
	at org.apache.http.impl.nio.DefaultNHttpClientConnection.consumeInput(DefaultNHttpClientConnection.java:265)
	at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:81)
	at org.apache.http.impl.nio.client.InternalIODispatch.onInputReady(InternalIODispatch.java:39)
	at org.apache.http.impl.nio.reactor.AbstractIODispatch.inputReady(AbstractIODispatch.java:114)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.readable(BaseIOReactor.java:162)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvent(AbstractIOReactor.java:337)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.processEvents(AbstractIOReactor.java:315)
	at org.apache.http.impl.nio.reactor.AbstractIOReactor.execute(AbstractIOReactor.java:276)
	at org.apache.http.impl.nio.reactor.BaseIOReactor.execute(BaseIOReactor.java:104)
	at org.apache.http.impl.nio.reactor.AbstractMultiworkerIOReactor$Worker.run(AbstractMultiworkerIOReactor.java:588)

But this code works well with ES v2.4. The request json payload seems to be correct because it is executed properly by sense.

POST /beam/test/_bulk?refresh=true
{ "index" : {} }
{"scientist":"Einstein", "id":0}
{ "index" : {} }
{"scientist":"Darwin", "id":1}

I did not see any breaking change in the website about this. What am I doing wrong?

  /** Inserts the given number of test documents into Elasticsearch. */
  static void insertTestDocuments(String index, String type, long numDocs, RestClient restClient)
      throws IOException {
    List<String> data =
        ElasticSearchIOTestUtils.createDocuments(
            numDocs, ElasticSearchIOTestUtils.InjectionMode.DO_NOT_INJECT_INVALID_DOCS);
    StringBuilder bulkRequest = new StringBuilder();
    for (String document : data) {
      bulkRequest.append(String.format("{ \"index\" : {} }%n%s%n", document));
    }
    String endPoint = String.format("/%s/%s/_bulk", index, type);
    HttpEntity requestBody =
        new NStringEntity(bulkRequest.toString(), ContentType.APPLICATION_JSON);
    Response response = restClient.performRequest("POST", endPoint,
        Collections.singletonMap("refresh", "true"), requestBody,
        new BasicHeader("", ""));
    JsonNode searchResult = ElasticsearchIO.parseResponse(response);
    boolean errors = searchResult.path("errors").asBoolean();
    if (errors){
      throw new IOException(
          String.format("Failed to insert test documents in index %s", index));
    }
  }


  /**
   * Generates a list of test documents for insertion.
   *
   * @param numDocs Number of docs to generate
   * @param injectionMode {@link InjectionMode} that specifies whether to insert malformed documents
   * @return the list of json String representing the documents
   */
  static List<String> createDocuments(long numDocs, InjectionMode injectionMode) {
    String[] scientists = {
      "Einstein",
      "Darwin",
      "Copernicus",
      "Pasteur",
      "Curie",
      "Faraday",
      "Newton",
      "Bohr",
      "Galilei",
      "Maxwell"
    };
    ArrayList<String> data = new ArrayList<>();
    for (int i = 0; i < numDocs; i++) {
      int index = i % scientists.length;
      // insert 2 malformed documents
      if (InjectionMode.INJECT_SOME_INVALID_DOCS.equals(injectionMode) && (i == 6 || i == 7)) {
        data.add(String.format("{\"scientist\";\"%s\", \"id\":%d}", scientists[index], i));
      } else {
        data.add(String.format("{\"scientist\":\"%s\", \"id\":%d}", scientists[index], i));
      }
    }
    return data;
  }
2

The call to restClient#performRequest should not contain new BasicHeader("", ""). If headers are empty you don't have to provide a fake one and you can directly call:

HttpEntity requestBody =
        new NStringEntity(bulkRequest.toString(), ContentType.APPLICATION_JSON);
    Response response = restClient.performRequest("POST", endPoint,
        Collections.singletonMap("refresh", "true"), requestBody);

In 5.4 when you set an empty header it fails with the following message:
empty headers are not allowed []

.. and it works when the header is removed.

3

Thanks Jim! Indeed it works on both ES versions when removing the empty headers.