Server returned HTTP response code: 400 for URL, when sending a post request to elasticsearch?

Kulasangar_Gowrisang · October 31, 2016, 7:44am

I'm using elasticsearch 5.0.0, and I've got three indexes which I'm trying to query and get data through my code. I'm sending a post request which includes a json body as well. It returns a 400 when when I'm running my code, but then when I just copy and pasted the url in the browser it return the results properly.

The code perfectly works when I tried using elasticsearch 2.3.5 and the it returns the data properly. I just couldn't find where am I going wrong. I checked the elasticsearch log as well, and there's no errors as such.

Could it be the version difference? Here is my code to send the post request:

public JSONObject getResult(String query, String elasticSearchIndex, String body) throws IOException, JSONException {
            if (query != null) {
                URL mUrl = new URL(getElasticSearchQueryUrlForEmptyQuery(query, elasticSearchIndex));
                HttpURLConnection conn = (HttpURLConnection) mUrl.openConnection();
                conn.setDoOutput(true);
                conn.setRequestMethod("POST");
                conn.setRequestProperty("Content-Type", "application/json");
    
                String input = body;
    
                OutputStream os = conn.getOutputStream();
                os.write(input.getBytes());
                os.flush();
    
                BufferedReader inputStream = new BufferedReader(new InputStreamReader(conn.getInputStream()));
                String inputLine;
                StringBuffer response = new StringBuffer();
    
                while ((inputLine = inputStream.readLine()) != null) {
                    response.append(inputLine);
                }
                return new JSONObject(response.toString());
            }
            return null;
        }

In the above I'm sending the json body for the request from another class.

And the body looks like this:

 {  
   "query":{  
      "query_string":{  
         "query":"myquery"
      }
   },
   "aggs":{  
      "total":{  
         "terms":{  
            "field":"userid"
         },
         "aggs":{  
            "grades_count":{  
               "value_count":{ 
                  "script": { "inline": "doc['user'].value", "lang": "groovy" }
               }
            }
         }
      }
   }
}

This is the whole exception:

Exception in thread "main" java.io.IOException: Server returned HTTP
response code: 400 for URL:
http://hostmachine:9200/indexname/_search?

Where am I going wrong? Any help could be appreciated.

xavierfacq · October 31, 2016, 8:51am

Hi,

A suggestion: When you run your code on the 5.0.0 version, do you use 5.0.0 Elasticsearch Jars ?

bye,
Xavier

Kulasangar_Gowrisang · October 31, 2016, 9:02am

@xavierfacq nop, I'm really unaware of it.

Should I import any jars into my project ?

Thanks.

xavierfacq · October 31, 2016, 9:10am

When your project "discuss" with the new version 5.0.0, It may uses the new version of Elasticsearch API.
Somewhere in your project your have defined the Elasticsearch client version. You have to find this configuration,
maybe in a pom.xml or in you /lib/ directory, and replace with the 5.0.0 version.

For exemple in a pom.xml

    <dependency>
		<groupId>org.elasticsearch</groupId>
		<artifactId>elasticsearch</artifactId>
		<version>2.2.0</version>
	</dependency>

Else, copy Jars you can find in your Elasticsearch installation directory into your project library directory.

Kulasangar_Gowrisang · October 31, 2016, 9:37am

@xavierfacq No, I didn't have any dependency as such in my pom or either don't have the jars. And FYI I'm not using any elasticsearch client version.

I'm just sending it as an http post request with a json body, where the url includes the host machine url and the name of the index.

xavierfacq · October 31, 2016, 9:43am

oh, yes OK...

Did you try to replace with a GET method ?

You shoud use the client API

Kulasangar_Gowrisang · October 31, 2016, 9:51am

But then if it's a GET , I won't be able to append the json body isn't it?

Or you want me to just check the GET whether it's working?

Without using it, there's no option of getting though this?

xavierfacq · October 31, 2016, 9:57am

Did you try to execute your JSON with a curl command ?

Kulasangar_Gowrisang · October 31, 2016, 10:12am

@xavierfacq Nop, would that make any difference?

What I found was this. According to this I changed my aggs and it's working once I recreated the index using a PUT and then tried pushing data to that index.

Once I query, it's working now. But then without recreating the whole thing is there any way I could insert the "fielddata": true for the existing index I've got at the moment?

xavierfacq · October 31, 2016, 10:24am

I don't know fielddata...

Here is the method to use it : https://www.elastic.co/guide/en/elasticsearch/reference/current/fielddata.html#_enabling_fielddata_on_literal_text_literal_fields

Hope this help.

Kulasangar_Gowrisang · October 31, 2016, 10:58am

@xavierfacq thanks.

If it's a string value I can do this and get the results. But for some fields which are actually string in the db and I need to convert them to either double or int in elasticsearch aggs before I get the results.

I sent a PUT request in order to create the index with the following body:

{
  "mappings": {
    "type": {
      "properties": {
        "userid": {
          "type": "text",
          "fielddata": true
        },
         "responsecode": {
          "type": "text",
          "fielddata": true
        },"chargeamount": {
          "type": "text",
          "fielddata": true
        }
      }
    }
  }
}

Currently I'm doing the below in order to convert string to int for the field (chargeamount) but then, in the result (chargeamount) still shows as a string value where as it should be a double. How can I define it within the aggs or within the mappings when I'm creating the index?

Where am I going wrong?

xavierfacq · October 31, 2016, 11:38am

If you want to get aggregations on fields you need to set in your mapping:

          "a_text_field" : {
		"type": "string",
		"index": "not_analyzed"
          },

          "an_integer_field" : {
		"type": "integer"
          },

Kulasangar_Gowrisang · October 31, 2016, 11:50am

@xavierfacq yes I changed the mapping afterwards, and it looks like this, but then apart from the double, the others are working fine. :

{
  "mappings": {
    "type": {
      "properties": {
        "userid": {
          "type": "text",
          "fielddata": true
        },
         "responsecode": {
          "type": "integer"
        },"chargeamount": {
          "type": "double"
        }
      }
    }
  }
}

But still it doesn't give me a double result. Do I have to do the below within the aggs when I'm trying to query?

"script":{
            "inline" : "Double.parseDoubl(doc['chargeamount'].value)"
         }

Or could I just pass in doc['chargeamount'].value?

I tried both, and the first one throws an exception as such and the second gives the results but as a string not as a double.

xavierfacq · October 31, 2016, 12:43pm

You have pushed a new Mapping version and reindexed your datas to a new index having this mapping ?
The exception seems to say that it tries to convert the double into a String, so the field seems to be a double.

Is it a lang = groovy or painless in your query ?

Kulasangar_Gowrisang · October 31, 2016, 12:47pm

Thanks for the continuous support

yes that's right.

I thought I was trying to give a double type of mapping for the field so that I can directly get the value automatically converted from string to double. So the exception comes from the first scenario when I do this:

    "script":{
                "inline" : "Double.parseDoubl(doc['chargeamount'].value)"
             }

But the below doesn't give any exception as such, and gives the result as a string but not double:

 "script":{
                "inline" : "doc['chargeamount'].value"
             }

I haven't specified any but as I remember I made the change script.inline: on in my elasticsearch.yml. Seems like groovy has been deprecated in v5.0.0 with painless. So which means I've got to mention "lang": "painless" within the script?

My querying goes like this:

{  
   "query":{  
      "query_string":{  
         "query":"myquery"
      }
   },
   "aggs":{  
      "total":{  
         "terms":{  
            "field":"userid"
         },
         "aggs":{  
            "total":{  
               "sum":{  
                   "script":{
                   	"inline" : "doc['chargeamount'].value"
                   }
               }
            }
         }
      }
   }
}

xavierfacq · October 31, 2016, 1:19pm

I'll not be able to help you more with Painless. I can only suggest to try some following expressions :

cf: https://www.elastic.co/guide/en/elasticsearch/reference/master/modules-scripting-painless.html#_accessing_doc_values_from_painless

"inline" : "double d = doc['chargeamount'].value; return d;"

"inline" : "Double.valueOf(doc['chargeamount'].value);"

"inline" : "Double.valueOf(doc['chargeamount'].value).intValue;"

Or some else to determine the type of your field... Good luck !

Kulasangar_Gowrisang · October 31, 2016, 4:31pm

@xavierfacq It was my mistake, I had a logstash conversion within the mutate filter in order to convert the field into double. I commented it out and tried with the existing configurations, it worked.

For whomever it might help. So had this is as my mapping:

 {
  "mappings": {
    "type": {
      "properties": {
        "userid": {
          "type": "text",
          "fielddata": true
        },
         "responsecode": {
          "type": "integer"
        },"chargeamount": {
          "type": "double"
        }
      }
    }
  }
}

And the query looks like this:

"script":{
            "inline" : "doc['chargeamount'].value"
         }