_bulk API not working when we migrate from ES 2.3 to ES 5.5

Elastic Stack Elasticsearch
1

We are trying to migrate from AWS elasticsearch 2.3 to 5.5 version with minimal code changes. Currently, we are using bulk api for 2.3 version and am able to successfully insert data into ES.

But, with the same api, I am unable to see transactions in kibana using ES 5.5 version.We are using Java ES client for this operation through mulesoft.

From the elasticsearch documents, I do not see any difference in how data is being ingested to ES using _bulk api between ES 2.3 and 5.5 versions. Am I missing something here? Please help me out.

ErrorERROR com.mulesoft.safedeque.AnalyticsBundler: error sending analytics org.mule.module.http.internal.request.ResponseValidatorException: Response code 400 mapped as failure.

2

Do you have a more detailed stacktrace?
Anything in elasticsearch logs ?

3

Hi David,

Unfortunately, the only error I see in mule console is

ErrorERROR com.mulesoft.safedeque.AnalyticsBundler: error sending analytics org.mule.module.http.internal.request.ResponseValidatorException: Response code 400 mapped as failure.

Regards
Keerthi

4

It’s a Mule error here. We can’t help.
I’m surprised there is no log in elasticsearch though.

5

In mule, we are connecting to analytics jar file. So, the error is actually originating from the Java file itself. If nothing is changing from ES 2.3 to ES 5.5, we should ideally not get any error in sending analytics. The only thing we changed in the ES host.

6

I think I have no idea of what you are talking about. Sorry but that's probably because I don't know Mule.

we are connecting to analytics jar file.

What's that?

So, the error is actually originating from the Java file itself.

Which file?

The only thing we changed in the ES host.

So you are trying to send data to a 5.5 cluster but with a client which has been designed for 2.3?

I'm sorry but do you have any link or content which describes exactly what you are doing?
Unless other knows...

7

I really apologize for not giving the complete context. Will try to clarify.

We built our API's on Mulesoft platform and asynchronously send the logs to a jar file. In this java file, we send the multiple log files to elasticsearch using bulk api. The elasticsearch cluster to which the data is being ingested is currently on 2.3 version. We have the data in elasticsearch and we are able to see the data in kibana as well.

Today, we spinned another ES cluster which is on version 5.5 and updated the Java code to reflect this change to new URL. This is the only change we did and now, we are unable to see the data in the new Elasticsearch cluster.

The only logs we have are the one's we listed above. I don't think it is related to mulesoft because, the issue happens only when the ES version is changing. It should be either something to do with the way we insert data using Elasticsearch Java client or the api we are using to push data to Elasticsearch.

8

What exactly did you do for

updated the Java code

?

9

We just updated the java code with the new ES 5.5 host name in the bulk api. Earlier it was pointing to ES 2.3 server.

10

You just change the hostname? That’s what you mean?

This Java code is using any other jar? Like an elasticsearch client? If so which one?

Can you share the code?

11

@dadoonet - By hostname, I mean we just changed the ES URL. Previous host was ES 2.3 and now it is ES 5.5. The jar file has other functionalities but acts like an es client.

Resolution:

We made the below two changes and the issue has been resolved.

  1. We are currently using query parameters at the end of the URL while querying ES 2.3. For ES 5.5, the query is failing when we use query parameters in the URL.
  2. We changed 'create' function to 'index' in the _bulk api when we migrated to ES 5.5.

Thank you for your inputs.

12

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.