400 Bad Request when trying to query elasticsearch using HttpClient

Elastic Stack Elasticsearch
1

I am trying to perform a query searching in elasticsearch using httpclient. Whenever I do so though, I get an error when hitting my HttpResponseMessage response variable. It tells me it is a 400 (Bad request).

I checked my jsonQuery and copy that jsonObject and paste it into the CLI of Kibana and I actually get results in the CLI. So I am not understanding why I am getting a 400 bad request.

        public async Task<string> GetVirtualSupport()
        {

            string jsonFromFile;

            using (var reader = new StreamReader(path))
            {
                jsonFromFile = reader.ReadToEnd();
            }

            var json = JsonConvert.DeserializeObject<VirtualSupport>(jsonFromFile);


           foreach (var item in json.Systems.System.Applications)
            {

                foreach (var x in item.Application.BusinessProcessSteps)
                {

                    foreach (var y in x.BusinessProcessStep.LogDataSources)
                    {
                        var fieldName = y.LogDataSource.LogFieldsMapping.LevelField.FieldName;
                        var fieldValue = y.LogDataSource.LogFieldsMapping.LevelField.FieldValue;
                    }
                }
            }

            var query = "{\"size\": 1000,\"query\": {\"bool\": {\"should\":[ {\"match\": { \"level\": \"Information\" } }, {\"match\": { \"level\": \"Error\" } } ], " +
                        "\"filter\": [ { \"range\": { \"@timestamp\": { \"gte\": \"2021-07-26T07:58:45.304-05:00\", \"lt\": \"2021-07-26T08:58:45.305-05:00\" } } } ]," +
                        "\"minimum_should_match\": 1 } } }";


            HttpClient client = new HttpClient();
            client.BaseAddress = new Uri("http://localhost:9200/");
            client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
            var contentData = new StringContent(query, Encoding.UTF8, "application/json");
            var response = client.PostAsync("customer-simulation-es-app-logs-development-2021-07/_search", contentData).Result;




            if (response.IsSuccessStatusCode)
                {  
                    var contents = response.Content.ReadAsStringAsync().Result;

                    var jsonData = JsonConvert.SerializeObject(contents);

                    Console.WriteLine(jsonData);
                    
                    return jsonData;
            }
            else
            {
                Console.WriteLine("{0} ({1}) \n {2}", (int)response.StatusCode, response.ReasonPhrase, response.RequestMessage);
                return response.ToString();
            }
            
        }

In the console.exe I get:

{
  "size": 1000,
  "query": {
    "bool": {
      "should": [
        {
          "match": {
            "level": "Information"
          }
        },
        {
          "match": {
            "level": "Error"
          }
        }
      ],
      "filter": [
        {
          "range": {
            "@timestamp": {
              "gte": "2021-07-26T07:58:45.304-05:00",
              "lt": "2021-07-26T08:58:45.305-05:00"
            }
          }
        }
      ],
      "minimum_should_match": 1
    }
  }
}

400 (Bad Request)
 Method: POST, RequestUri: 'http://localhost:9200/customer-simulation-es-app-logs*/_search', Version: 1.1, Content: System.Net.Http.Json.JsonContent, Headers:
{
  Accept: application/json
  Transfer-Encoding: chunked
  Content-Type: application/json; charset=utf-8
}

Could it be a problem with the .PostAsJsonAsync()? If I am wanting to retrieve the results would that be the correct way? Not sure what is causing me to get a 400 bad request.

2

Hi,

PostAsJsonAsync will serialise the object you pass to it as JSON, so in this case, it's serialising the JObject itself, which is not what you want here. There are a few solutions, but since you already have a string representing your JSON, you can send that as StringContent via the PostAsync method on HttpClient. Using strings for the request and response may not be the most optimal route for performance but will work fine with a few tweaks to your sample. Something like this should work:

public async Task<string> GetVirtualSupportAsync()
{
	var query = "{\"size\": 1000,\"query\": {\"bool\": {\"should\":[ {\"match\": { \"level\": \"Information\" } }, {\"match\": { \"level\": \"Error\" } } ], " +
				"\"filter\": [ { \"range\": { \"@timestamp\": { \"gte\": \"2021-07-26T07:58:45.304-05:00\", \"lt\": \"2021-07-26T08:58:45.305-05:00\" } } } ]," +
				"\"minimum_should_match\": 1 } } }";

	HttpClient client = new HttpClient();
	client.BaseAddress = new Uri("http://localhost:9200/");
	client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
	HttpResponseMessage response = await client.PostAsync("customer-simulation-es-app-logs*/_search", new StringContent(query, Encoding.UTF8, "application/json"));

	if (response.IsSuccessStatusCode)
	{
		var contents = await response.Content.ReadAsStringAsync();
		Console.WriteLine(contents);
		return contents;
	}
	else
	{
		Console.WriteLine("{0} ({1}) \n {2}", (int)response.StatusCode, response.ReasonPhrase, response.RequestMessage);
		return response.ToString();
	}
}

Note that I also changed the code to use the async/await keywords as calling .Result in code can be problematic.

The response contents in this case will be the raw JSON string from the server, so you will then need to deserialise that or parse the data you require. To avoid a lot of this manual code, have you considered using our official .NET client library? This allows you to work with strongly-typed requests and responses, while also handling the HTTP transport and serialisation efficiently.

3

Hi @stevejgordon ,

Thank you for providing me with your help. That would make a lot of sense why PostAsJsonAsync() was not allowing me to perform the request. Sorry, I am a bit confused. If the variable response is a JSON string, shouldn't we need to serialize it into a JSON object not deserialise it since it is already a string? Please correct me if I am wrong. Yes, I have used NEST prior to this but just wanted to just use httpclient. I know it not the most ideal way in doing this.

On another note, is there a way to use string interpolation to replace certain parts of the query variable without having to create a class?

I am trying to replace some parts of my query with a variable that holds a certain value.

For instance, inside of my variable query I am wanting to replace

  • \"level\": \"Information\"
  • and \"match\": { \"level\": \"Error\"
  • with variable fieldName and variable fieldValue respectively
    • The fieldName holds a value of "_source.level"
    • and fieldValue holds a value of "information" and "error".

Snippet of the json file that I am reading from and assigning fieldname and fieldvalue to.

"LogFieldsMapping": {
	"IDField": { "FieldName": "_id" },
	"LevelField": {
		"FieldName": "_source.level",
		"FieldValue": [
			{ "value": "Information" },
			{ "value": "Error" }
		]
	}
}

The issue I am running into is normally I know you would add a $ before the string and add a {} where you want to call the variable. However, when I add a $ at the beginning of my string it does not like it and says

Only assignment, call, increment, decrement, await, and new object expressions can be used as a statement

So I am facing issues trying to figure out how I can pass those variables into my query variable. I will edit the original post. I am reading from a JSON file and assigning a variable to those values that I want to grab and insert inside of the query variable. So I am just wanting to use string interpolation in my case and not create a class.

4

My apologies for the confusion I've caused. I mistyped! The response is a HttpResponseMessage, it's the contents variable which is a string containing the JSON, the result of calling .Content.ReadAsStringAsync().

The issue you raise around replacing parts of the query in the string form is one of the reasons I wouldn't recommend this code for heavy production use. Maintaining the JSON manually can be quite brittle. If you must use that approach, you may want to look at using a StringBuilder to form the string, appending literal text and your variables as necessary. It will just be a pain to maintain it that way. If you want to use string interpolation, you will need to escape the curly braces which can be achieved by using two braces. Those are then treated literally. For example...

$"{{\"size\": 1000,\"query\": {{\"bool...

This obviously becomes complicated to read, quite quickly.

NEST makes this a lot easier as one can work with strongly-typed requests so one can simply update properties with any variables which are serialised.

5

@stevejgordon Oh no worries, it's okay! Thank you for the clarification. So Since the contents is a string containing the JSON, result of calling .Content.ReadAsStringAsync() I would need to deserialize the string back into an JSON object correct as var jsonData = JsonConvert.DeserializeObject(contents)?

Yes, I noticed how much more difficult it is by using HttpClient rather than it is to use NEST. But thank you for helping me with this. I appreciate your help and explanation!

6

You're welcome. Yes, once you have the JSON, you can deserialise it into an object representing the data. Note that the search response object can get complex quite quickly due to aggregations in particular.

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.