Server returned HTTP response code: 400 for URL, when sending a post request to elasticsearch?

[Kulasangar_Gowrisang (Kulasangar Gowrisangar)]

I'm using elasticsearch 5.0.0, and I've got three indexes which I'm trying to query and get data through my code. I'm sending a post request which includes a json body as well. It returns a 400 when when I'm running my code, but then when I just copy and pasted the url in the browser it return the results properly.

The code perfectly works when I tried using elasticsearch 2.3.5 and the it returns the data properly. I just couldn't find where am I going wrong. I checked the elasticsearch log as well, and there's no errors as such.

Could it be the version difference? Here is my code to send the post request:

public JSONObject getResult(String query, String elasticSearchIndex, String body) throws IOException, JSONException {
            if (query != null) {
                URL mUrl = new URL(getElasticSearchQueryUrlForEmptyQuery(query, elasticSearchIndex));
                HttpURLConnection conn = (HttpURLConnection) mUrl.openConnection();
                conn.setDoOutput(true);
                conn.setRequestMethod("POST");
                conn.setRequestProperty("Content-Type", "application/json");
    
                String input = body;
    
                OutputStream os = conn.getOutputStream();
                os.write(input.getBytes());
                os.flush();
    
                BufferedReader inputStream = new BufferedReader(new InputStreamReader(conn.getInputStream()));
                String inputLine;
                StringBuffer response = new StringBuffer();
    
                while ((inputLine = inputStream.readLine()) != null) {
                    response.append(inputLine);
                }
                return new JSONObject(response.toString());
            }
            return null;
        }

In the above I'm sending the json body for the request from another class.

And the body looks like this:

 {  
   "query":{  
      "query_string":{  
         "query":"myquery"
      }
   },
   "aggs":{  
      "total":{  
         "terms":{  
            "field":"userid"
         },
         "aggs":{  
            "grades_count":{  
               "value_count":{ 
                  "script": { "inline": "doc['user'].value", "lang": "groovy" }
               }
            }
         }
      }
   }
}

This is the whole exception:

Exception in thread "main" java.io.IOException: Server returned HTTP
response code: 400 for URL:
http://hostmachine:9200/indexname/_search?

Where am I going wrong? Any help could be appreciated.

[xavierfacq]

Hi,

A suggestion: When you run your code on the 5.0.0 version, do you use 5.0.0 Elasticsearch Jars ?

bye,
Xavier

[Kulasangar_Gowrisang (Kulasangar Gowrisangar)]

@xavierfacq nop, I'm really unaware of it.

Should I import any jars into my project ?

Thanks.

[xavierfacq]

When your project "discuss" with the new version 5.0.0, It may uses the new version of Elasticsearch API.
Somewhere in your project your have defined the Elasticsearch client version. You have to find this configuration,
maybe in a pom.xml or in you /lib/ directory, and replace with the 5.0.0 version.

For exemple in a pom.xml

    <dependency>
		<groupId>org.elasticsearch</groupId>
		<artifactId>elasticsearch</artifactId>
		<version>2.2.0</version>
	</dependency>

Else, copy Jars you can find in your Elasticsearch installation directory into your project library directory.

[Kulasangar_Gowrisang (Kulasangar Gowrisangar)]

@xavierfacq No, I didn't have any dependency as such in my pom or either don't have the jars. And FYI I'm not using any elasticsearch client version.

I'm just sending it as an http post request with a json body, where the url includes the host machine url and the name of the index.

[xavierfacq]

oh, yes OK...

@Kulasangar_Gowrisang

conn.setRequestMethod("POST");

Did you try to replace with a GET method ?

You shoud use the client API

[Kulasangar_Gowrisang (Kulasangar Gowrisangar)]

But then if it's a GET , I won't be able to append the json body isn't it?

Or you want me to just check the GET whether it's working?

Without using it, there's no option of getting though this?

[xavierfacq]

Did you try to execute your JSON with a curl command ?

[Kulasangar_Gowrisang (Kulasangar Gowrisangar)]

@xavierfacq Nop, would that make any difference?

What I found was this. According to this I changed my aggs and it's working once I recreated the index using a PUT and then tried pushing data to that index.

Once I query, it's working now. But then without recreating the whole thing is there any way I could insert the "fielddata": true for the existing index I've got at the moment?

[xavierfacq]

I don't know fielddata...

Here is the method to use it : https://www.elastic.co/guide/en/elasticsearch/reference/current/fielddata.html#_enabling_fielddata_on_literal_text_literal_fields

Hope this help.

[Kulasangar_Gowrisang (Kulasangar Gowrisangar)]

@xavierfacq thanks.

If it's a string value I can do this and get the results. But for some fields which are actually string in the db and I need to convert them to either double or int in elasticsearch aggs before I get the results.

I sent a PUT request in order to create the index with the following body:

{
  "mappings": {
    "type": {
      "properties": {
        "userid": {
          "type": "text",
          "fielddata": true
        },
         "responsecode": {
          "type": "text",
          "fielddata": true
        },"chargeamount": {
          "type": "text",
          "fielddata": true
        }
      }
    }
  }
}

Currently I'm doing the below in order to convert string to int for the field (chargeamount) but then, in the result (chargeamount) still shows as a string value where as it should be a double. How can I define it within the aggs or within the mappings when I'm creating the index?

Where am I going wrong?

[xavierfacq]

If you want to get aggregations on fields you need to set in your mapping:

          "a_text_field" : {
		"type": "string",
		"index": "not_analyzed"
          },

          "an_integer_field" : {
		"type": "integer"
          },

[Kulasangar_Gowrisang (Kulasangar Gowrisangar)]

@xavierfacq yes I changed the mapping afterwards, and it looks like this, but then apart from the double, the others are working fine. :

{
  "mappings": {
    "type": {
      "properties": {
        "userid": {
          "type": "text",
          "fielddata": true
        },
         "responsecode": {
          "type": "integer"
        },"chargeamount": {
          "type": "double"
        }
      }
    }
  }
} 

But still it doesn't give me a double result. Do I have to do the below within the aggs when I'm trying to query?

"script":{
            "inline" : "Double.parseDoubl(doc['chargeamount'].value)"
         } 

Or could I just pass in doc['chargeamount'].value?

I tried both, and the first one throws an exception as such and the second gives the results but as a string not as a double.

[xavierfacq]

You have pushed a new Mapping version and reindexed your datas to a new index having this mapping ?
The exception seems to say that it tries to convert the double into a String, so the field seems to be a double.

Is it a lang = groovy or painless in your query ?

[Kulasangar_Gowrisang (Kulasangar Gowrisangar)]

Thanks for the continuous support

@xavierfacq

You have pushed a new Mapping version and reindexed your datas to a new index having this mapping ?

yes that's right.

@xavierfacq

The exception seems to say that it tries to convert the double into a String, so the field seems to be a double.

I thought I was trying to give a double type of mapping for the field so that I can directly get the value automatically converted from string to double. So the exception comes from the first scenario when I do this:

    "script":{
                "inline" : "Double.parseDoubl(doc['chargeamount'].value)"
             }

But the below doesn't give any exception as such, and gives the result as a string but not double:

 "script":{
                "inline" : "doc['chargeamount'].value"
             } 

@xavierfacq

Is it a lang = groovy or painless in your query ?

I haven't specified any but as I remember I made the change script.inline: on in my elasticsearch.yml. Seems like groovy has been deprecated in v5.0.0 with painless. So which means I've got to mention "lang": "painless" within the script?

My querying goes like this:

{  
   "query":{  
      "query_string":{  
         "query":"myquery"
      }
   },
   "aggs":{  
      "total":{  
         "terms":{  
            "field":"userid"
         },
         "aggs":{  
            "total":{  
               "sum":{  
                   "script":{
                   	"inline" : "doc['chargeamount'].value"
                   }
               }
            }
         }
      }
   }
}

[xavierfacq]

I'll not be able to help you more with Painless. I can only suggest to try some following expressions :

cf: https://www.elastic.co/guide/en/elasticsearch/reference/master/modules-scripting-painless.html#_accessing_doc_values_from_painless

"inline" : "double d = doc['chargeamount'].value; return d;"

"inline" : "Double.valueOf(doc['chargeamount'].value);"

"inline" : "Double.valueOf(doc['chargeamount'].value).intValue;"

Or some else to determine the type of your field... Good luck !

[Kulasangar_Gowrisang (Kulasangar Gowrisangar)]

@xavierfacq It was my mistake, I had a logstash conversion within the mutate filter in order to convert the field into double. I commented it out and tried with the existing configurations, it worked.

For whomever it might help. So had this is as my mapping:

 {
  "mappings": {
    "type": {
      "properties": {
        "userid": {
          "type": "text",
          "fielddata": true
        },
         "responsecode": {
          "type": "integer"
        },"chargeamount": {
          "type": "double"
        }
      }
    }
  }
}

And the query looks like this:

"script":{
            "inline" : "doc['chargeamount'].value"
         }