Elasticsearch.Net 7.17.X unable to connect Elasticsearch 8.13.4 version

Elastic Stack Elasticsearch
1

I have upgraded from elasticsearch 2.3.2 to 8.13.4. I have created a multi node cluster and I am trying to connect to elasticsearch through my .NET. I have heard that elastic.client.elasticsearch library is not providing all the functionalities so the this blog Connecting to Elasticsearch v8.x using the v7.17.x client | Elasticsearch .NET Clients [7.17] | Elastic is asking to connect to 8.X using 7.17.X clients to access the data and indices. Below is my code:

var pool = new SingleNodeConnectionPool(new Uri(ConfigurationManager.AppSettings["ElasticSearchUrl"]));
     
        var setting = new Nest.ConnectionSettings(pool)
                          .CertificateFingerprint("adsfa.asdfasdfasfasdfasdfasdfasdfasdfasdfasdfasdfasfasdfasdfasf")
                          .RequestTimeout(TimeSpan.FromMinutes(2))
                          .BasicAuthentication(ConfigurationManager.AppSettings["elasticUser"], ConfigurationManager.AppSettings["elasticPwd"])
                          .EnableApiVersioningHeader();
        var client = new Nest.ElasticClient(setting);
        var lowlevelClient = client.LowLevel;
        string es_query = @"{
          ""query"": {
          ""_source"": [""Index""]
          ""query"": {
                        ""match_all"": {}
                    }
        }";

        Debug.WriteLine(es_query);
        try
        {

            
            var searchResponse = lowlevelClient.Search<BytesResponse>("demoindexfortest", PostData.String(es_query));
            var responseBytes = searchResponse.Body;
}

However, I am getting the below error while executing the search function:
{Unsuccessful () low level call on POST: /demoindexfortest/_search}

Could someone help so that I can get out of this issue?

2

Hi @sathish12,

I would strongly recommend trying out the 8.14.x version of Elastic.Clients.Elasticsearch. There have been a lot of improvements over the past months and nearly all functionality is supported now. Besides that, NEST is deprecated and will be end of life at the end of this year. Newer functionality of ES 8 is exclusively implemented in Elastic.Clients.Elasticsearch already.

However, I am getting the below error while executing the search function:
{Unsuccessful () low level call on POST: /demoindexfortest/_search}

Without the complete error response, it's hard to tell what's going on. From a first glance it seems like your JSON payload is invalid (you got 2 times query and at least one closing } is missing).

Below is my code

Is there any specific reason for using the low-level client and a hand-crafted JSON request? If you don't explicitly want to do this, the new Elastic.Client.Elasticsearch client can be used to execute the desired query:

var response = await client.SearchAsync<Person>(r => r
	.Index("demoindexfortest")
	.Query(q => q
		.MatchAll(ma => { })
	)
);

foreach (var document in response.Documents)
{
	Console.WriteLine(document.Id);
}
3

Hi flobernd,
Thanks for the the response. We have setup a multinode and multiple master eligible cluster. However, it took lot of time to setup this up in our machines. I am not sure how much time it would take to setup the cluster from the scratch with 8.14.x the latest. The problem is it was taking lot of time to resolve issue of split brain situation. Later I learnt that I have to delete the data, config/certs folders and change the elasticsearch.yml every time I add a new machine to the cluster. However, I am ready to migrate to 8.14.X if there is any easy way where I dont have to setup from the scratch and do not lose the data which is present right now.

I am extremely sorry as the above json provided is a typo. Below is the proper one which I have tried and getting the error:

However The response is the same

As I said flobernd, we are upgrading from 2.3.2 to 8.13.4. So our .NET code was using string based query at one point. and we are trying to use the same without changing. Here we have only provided basic string query. But we are using a big json query which has aggregations, filters on the date and fields.

4

You don't have to update the cluster in order to use Elastic.Clients.Elasticsearch 8.14.x over NEST. Only the client package must be changed.

Any chance you could post the complete error message / debug information? The error message you've provided is too generic to find out the root cause of this.

While using the low level client is of course an option, I would still recommend you to interatively update pieces of your code to use the high level API instead. It provides many quality of life improvements, type-safety and minimizes the overall risk of writing bad queries.

5

Sure we will update the clientband try it out. But any documentation on this elastic.client.elasticsearch would be really helpful

6

Hi @flobernd ,
We have upgraded our elastic.client.elasticsearch library to new 8.15.5 and we have a requirement to query one of the indices that we should get the data between a specific datetime range. Below is the json query which we have prepared while testing through curl

{
    "query": {
        "bool": {
            "must": [
            {
                "query_string": {
                "analyze_wildcard": true,
                "query": "*"
                }
            },
            {
                "range": {
                    "@timestamp": {
                        "gte": 1724602611000,
                        "lte": 1724607616000,
                        "format": "epoch_millis"
                    }
                }
            }
            ],
            "must_not": []
        }
        },
        "size": 0,
        "aggs": {
        "timev": {
            "date_histogram": {
            "field": "@timestamp",
            "fixed_interval": "1m", 
            "min_doc_count": 1,
            "extended_bounds": {
                "min": 1724602611000,
                "max": 1724607616000
            }
            },
            "aggs": {
            "avv": {
                "avg": {
                "field": "demo_field_name"
                }
            }
            }
        }
        }
}

and we are getting our data with the above json query. However, we are not able to convert this query to .NET client code. we are seeing multiple method not found errors while trying to write the code. Is there any way where we can pass the raw json or write the code with ease. Because we are new to this and I dont find the proper documentation also on the code base

7

Hi @sathish12,

the DSL of the new client maps pretty closely to the JSON structure and should allow for a nearly 1:1 translation.

Which methods are missing for you?

Edit: I took the time to translate the query for you, hoping that this will get you started. As you can see, besides of the C# specific things, the DSL pretty closely matches the structure of the JSON request.

var re = await client.SearchAsync<MyDocument>(s => s
    .Query(q => q
        .Bool(b => b
            .Must(
            [
                q => q.QueryString(qs => qs
                    .AnalyzeWildcard(true)
                    .Query("*")
                ),
                q => q.Range(rq => rq
                    .DateRange(dr => dr
                        .Field("@timestamp"!)
                        .Gte(DateMath.Anchored(DateTimeOffset.FromUnixTimeMilliseconds(1724602611000).DateTime))
                        .Lte(DateMath.Anchored(DateTimeOffset.FromUnixTimeMilliseconds(1724607616000).DateTime))
                        .Format("epoch_millis")
                    )
                )
            ])
            .MustNot(new Query[] { })
        )
    )
    .Size(0)
    .Aggregations(aggs => aggs
        .Add("timev", agg => agg
            .DateHistogram(dh => dh
                .Field("@timestamp"!)
                .FixedInterval(TimeSpan.FromMinutes(1))
                .MinDocCount(1)
                .ExtendedBounds(bnd => bnd
                    .Min(new FieldDateMath(1724602611000))
                    .Max(new FieldDateMath(1724607616000))
                )
            )
            .Aggregations(sub => sub
                .Add("avv", agg => agg
                    .Avg(aa => aa
                        .Field("demo_field_name")
                    )
                )
            )
        )
    )
);