Elasticsearch is very slow

I set up Elastic Search about a month ago for my pet-project. Elastic Search is hosted locally on my machine on Windows. Everything was working fine. Now when I try to access it in any way it doesn't respond and I get an error that the request timeout has been exceeded. I also tried completely reinstalling it and now it's empty, the cluster is green, but it's still not responding

At first I thought something had happened to the local server and it wouldn't start, but I can follow the link and it pings.

I also tried via postman to create an index, but also nothing happens for 10 minutes

This is what I get when I go to the link in my browser:

{
    "name": "MachineName",
    "cluster_name": "elasticsearch",
    "cluster_uuid": "Fodf_SsVSG2weJ39uSX5lA",
    "version": {
        "number": "8.17.0",
        "build_flavor": "default",
        "build_type": "zip",
        "build_hash": "2b6a7fed44faa321997703718f07ee0420804b41",
        "build_date": "2024-12-11T12:08:05.663969764Z",
        "build_snapshot": false,
        "lucene_version": "9.12.0",
        "minimum_wire_compatibility_version": "7.17.0",
        "minimum_index_compatibility_version": "7.0.0"
    },
    "tagline": "You Know, for Search"
}

That's the response I'm getting when I'm pinging:

Ping statistics for 192.168.1.19:
    Packets: sent = 4, received = 4, lost = 0
    (0% loss).
Approximate receive-transmit time in ms:
    Minimum = 0msec, Maximum = 0 msec, Average = 0 msec

But still any action always exceeds the timeout, even at 5 minutes

Here's the code for my ElasticService

using Elastic.Clients.Elasticsearch;
using Elastic.Clients.Elasticsearch.QueryDsl;
using Elastic.Transport;
using Microsoft.Extensions.Logging;
using Microsoft.Extensions.Options;
using PriceScout.Server.Application.Services.Interfaces;
using PriceScout.Server.Domain.Products;
using PriceScout.Server.Infrastructure.Settings;

namespace PriceScout.Server.Infrastructure.Services
{
    public class ElasticService : IElasticService
    {
        private readonly ILogger<ElasticService> _logger;
        private readonly ElasticsearchClient _elasticClient;
        private readonly ElasticSettings _elasticSettings;

        public ElasticService(IOptions<ElasticSettings> options, ILogger<Services.ElasticService> logger)
        {
            _elasticSettings = options.Value;
            var settings = new ElasticsearchClientSettings(new Uri(_elasticSettings.Url))
                .Authentication(new BasicAuthentication(_elasticSettings.Username, _elasticSettings.Password)) 
                .RequestTimeout(TimeSpan.FromMinutes(1)).MaximumRetries(3).MaxRetryTimeout(TimeSpan.FromMinutes(1)).EnableDebugMode();
            ;
            _elasticClient = new ElasticsearchClient(settings);
            _logger = logger;
        }

        public async Task CreateIndex(CancellationToken cancellationToken = default)
        {
            // Check if the index already exists
            var existsResponse = await IndexExistsAsync("products", cancellationToken);
            if (!existsResponse)
            {
                var createIndexResponse = await _elasticClient.Indices.CreateAsync<Product>("products", p => p
                    .Mappings(map => map
                        .Properties(prop => prop
                            .Keyword(p => p.SKU)
                            .Keyword(p => p.CactusArticleNumber)
                            .Keyword(p => p.DarwinArticleNumber)
                            .Keyword(p => p.EnterArticleNumber)
                            .Keyword(p => p.UltraArticleNumber)
                            .Object(p => p.Category, obj => obj
                                .Properties(ps => ps
                                    .Text(t => t.Category.NameRu)
                                    .Text(t => t.Category.NameEn)
                                    .Text(t => t.Category.NameRo)
                                    .Text(t => t.Category.DescriptionRu)
                                    .Text(t => t.Category.DescriptionEn)
                                    .Text(t => t.Category.DescriptionRo)
                                )
                            )
                            .Nested(p => p.Localizations, n => n
                                .Properties(p => p
                                .Text(l => l.Localizations.First().TranslatedName)
                                .Text(l => l.Localizations.First().TranslatedDescription)
                                .Text(l => l.Localizations.First().TranslatedShortDescription)
                            )
                        )
                    )
                )
                );

                if (!createIndexResponse.IsValidResponse)
                {
                    _logger.LogError("Failed to create index 'products':{error}", createIndexResponse.ElasticsearchServerError?.Error.Reason);
                    throw new Exception($"Failed to create index 'products': {createIndexResponse.ElasticsearchServerError?.Error.Reason}");
                }

                _logger.LogInformation("Index created successfully");
            }
        }

        public async Task<bool> IndexExistsAsync(string indexName, CancellationToken cancellationToken = default)
        {
            var existsResponse = await _elasticClient.Indices.ExistsAsync(indexName, cancellationToken);
            return existsResponse.Exists;
        }

        public async Task<bool> AddOrUpdateAsync(Product product, CancellationToken cancellationToken = default)
        {
            var response = await _elasticClient.IndexAsync(product, i => i.Index("products").Refresh(Elastic.Clients.Elasticsearch.Refresh.False), cancellationToken);
            return response.IsValidResponse;
        }

        public async Task<bool> BulkAddOrUpdateAsync(IEnumerable<Product> products, CancellationToken cancellationToken = default)
        {
            var bulkResponse = await _elasticClient.BulkAsync(b => b.Index("products")
                                                                        .UpdateMany(products, (ud, p) => ud.Doc(p).DocAsUpsert(true)), cancellationToken);

            _logger.LogInformation($"Bulk AddOrUpdate successfully!");
            return bulkResponse.IsValidResponse && !bulkResponse.Errors;
        }

        public async Task<Product> GetByIdAsync(string id, CancellationToken cancellationToken = default)
        {
            var response = await _elasticClient.GetAsync<Product>(id, g => g.Index("products"), cancellationToken);
            if (response.IsValidResponse && response.Found)
            {
                return response.Source;
            }
            return null;
        }

        public async Task<IEnumerable<int>> SearchAsync(
            string searchTerm,
            int from = 0,
            int take = 5,
            int size = 10, CancellationToken cancellationToken = default
        )
        { 
            var response = await _elasticClient.SearchAsync<Product>(s => s
            .Index("products")
            .SourceIncludes("id")
            .Query(q => q
                .Bool(b => b
                    .Should(
                        bs => bs.Term(t => t.Field(f => f.SKU).Value(searchTerm).Boost(10)),
                        bs => bs.Bool(bb => bb
                            .Should(
                                bs1 => bs1.Term(t => t.Field(f => f.DarwinArticleNumber).Value(searchTerm).Boost(9)),
                                bs2 => bs2.Term(t => t.Field(f => f.EnterArticleNumber).Value(searchTerm).Boost(9)),
                                bs3 => bs3.Term(t => t.Field(f => f.CactusArticleNumber).Value(searchTerm).Boost(9)),
                                bs4 => bs4.Term(t => t.Field(f => f.UltraArticleNumber).Value(searchTerm).Boost(9))
                            )
                        ),
                        bs => bs.Nested(n => n
                            .Path(p => p.Localizations)
                            .Query(nq => nq
                                .MultiMatch(mm => mm
                                    .Query(searchTerm)
                                    .Fields("localizations.translatedName")
                                    .Fuzziness(new Fuzziness("auto"))
                                    .Operator(Operator.Or)
                                )
                            )
                        ),
                        bs => bs.MultiMatch(mm => mm
                            .Query(searchTerm)
                            .Fields("category.nameRu,category.nameEn,category.nameRo")
                            .Fuzziness(new Fuzziness("auto"))
                            .Operator(Operator.Or)
                            .Boost(2)
                        ),
                        bs => bs.Bool(bb => bb
                            .Should(
                                bs1 => bs1.Nested(n => n
                                    .Path(p => p.Localizations)
                                    .Query(nq => nq
                                        .MultiMatch(mm => mm
                                            .Query(searchTerm)
                                            .Fields("localizations.translatedDescription, localizations.translatedShortDescription")
                                            .Fuzziness(new Fuzziness("auto"))
                                            .Operator(Operator.Or)
                                        )
                                    )
                                )
                            )
                            .Boost(1)
                        ),
                        bs => bs.MultiMatch(mm => mm
                            .Query(searchTerm)
                            .Fields("category.descriptionRu, category.descriptionEn, category.descriptionRo")
                            .Fuzziness(new Fuzziness("auto"))
                            .Operator(Operator.Or)
                            .Boost(1)
                        )
                    )
                    .MinimumShouldMatch(1)
                )
            )
            .Sort(srt => srt.Score(new ScoreSort()) 
            ), cancellationToken
        );


            if (!response.IsValidResponse)
            {
                throw new Exception($"Search request failed: {response.ElasticsearchServerError?.Error.Reason}");
            }

            return response.Documents.Take(take).Select(d => d.Id).ToList();
        }

        public async Task<bool> DeleteByIdAsync(string id, CancellationToken cancellationToken = default)
        {
            var response = await _elasticClient.DeleteAsync<Product>(id, d => d.Index("products"), cancellationToken);
            return response.IsValidResponse;
        }
    }
}

The error I'm getting:

Invalid Elasticsearch response built from a unsuccessful () low level call on PUT: /products?pretty=true&error_trace=true
 Exception: The request was canceled due to the configured HttpClient.Timeout of 60 seconds elapsing.

# Audit trail of this API call:
 - [1] BadRequest: Node: http://192.168.1.19:9200/ Took: 00:01:01.5998508
 - [2] MaxTimeoutReached: Took: 00:00:00.0000007
# OriginalException: System.Threading.Tasks.TaskCanceledException: The request was canceled due to the configured HttpClient.Timeout of 60 seconds elapsing.
 ---> System.TimeoutException: The operation was canceled.
 ---> System.Threading.Tasks.TaskCanceledException: The operation was canceled.
 ---> System.IO.IOException: Unable to read data from the transport connection: Операция ввода/вывода была прервана из-за завершения потока команд или по запросу приложения..
 ---> System.Net.Sockets.SocketException (995): Операция ввода/вывода была прервана из-за завершения потока команд или по запросу приложения.
   --- End of inner exception stack trace ---
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.ThrowException(SocketError error, CancellationToken cancellationToken)
   at System.Net.Sockets.Socket.AwaitableSocketAsyncEventArgs.System.Threading.Tasks.Sources.IValueTaskSource<System.Int32>.GetResult(Int16 token)
   at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpConnection.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpConnectionPool.SendWithVersionDetectionAndRetryAsync(HttpRequestMessage request, Boolean async, Boolean doRequestAuth, CancellationToken cancellationToken)
   at System.Net.Http.RedirectHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.DecompressionHandler.SendAsync(HttpRequestMessage request, Boolean async, CancellationToken cancellationToken)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   --- End of inner exception stack trace ---
   --- End of inner exception stack trace ---
   at System.Net.Http.HttpClient.HandleFailure(Exception e, Boolean telemetryStarted, HttpResponseMessage response, CancellationTokenSource cts, CancellationToken cancellationToken, CancellationTokenSource pendingRequestsCts)
   at System.Net.Http.HttpClient.<SendAsync>g__Core|83_0(HttpRequestMessage request, HttpCompletionOption completionOption, CancellationTokenSource cts, Boolean disposeCts, CancellationTokenSource pendingRequestsCts, CancellationToken originalCancellationToken)
   at Elastic.Transport.HttpRequestInvoker.RequestCoreAsync[TResponse](Boolean isAsync, Endpoint endpoint, BoundConfiguration boundConfiguration, PostData postData, CancellationToken cancellationToken)
# Request:
{
  "mappings": {
    "properties": {
      "sku": {
        "type": "keyword"
      },
      "cactusArticleNumber": {
        "type": "keyword"
      },
      "darwinArticleNumber": {
        "type": "keyword"
      },
      "enterArticleNumber": {
        "type": "keyword"
      },
      "ultraArticleNumber": {
        "type": "keyword"
      },
      "category": {
        "properties": {
          "nameRu": {
            "type": "text"
          },
          "nameEn": {
            "type": "text"
          },
          "nameRo": {
            "type": "text"
          },
          "descriptionRu": {
            "type": "text"
          },
          "descriptionEn": {
            "type": "text"
          },
          "descriptionRo": {
            "type": "text"
          }
        },
        "type": "object"
      },
      "localizations": {
        "properties": {
          "translatedName": {
            "type": "text"
          },
          "translatedDescription": {
            "type": "text"
          },
          "translatedShortDescription": {
            "type": "text"
          }
        },
        "type": "nested"
      }
    }
  }
}
# Response:
<Response stream not captured or already read to completion by serializer. Set DisableDirectStreaming() on TransportConfiguration to force it to be set on the response.>
# TCP states:
  Established: 30
  FinWait2: 2
  CloseWait: 4
  TimeWait: 6
  SynSent: 2

# ThreadPool statistics:
  Worker: 
    Busy: 2
    Free: 32765
    Min: 6
    Max: 32767
  IOCP: 
    Busy: 0
    Free: 1000
    Min: 1
    Max: 1000

ping isn't telling you anything about elasticsearch, elasticsearch does not responding to ICMP - all that ping is saying is the server is up and on the network and responding to ICMPs.

Do you have kibana installed? If so, what is it showing about your cluster (single node cluster!)

Can you do a simple GET on /_cluster/health or /_cat/indices ?

You said "Everything was working fine", can you elaborate? You were regularly indexing and analyzing data, applications were working, ... ?

I suspect something on the windows server itself, does the event log show anything?

/_cluster/health:

{
    "cluster_name": "elasticsearch",
    "status": "green",
    "timed_out": false,
    "number_of_nodes": 1,
    "number_of_data_nodes": 1,
    "active_primary_shards": 31,
    "active_shards": 31,
    "relocating_shards": 0,
    "initializing_shards": 0,
    "unassigned_shards": 0,
    "unassigned_primary_shards": 0,
    "delayed_unassigned_shards": 0,
    "number_of_pending_tasks": 0,
    "number_of_in_flight_fetch": 0,
    "task_max_waiting_in_queue_millis": 0,
    "active_shards_percent_as_number": 100.0
}

/_cat/indices

green open .internal.alerts-transform.health.alerts-default-000001            nfJnh64LQlmzMevLP1JJRw 1 0 0 0 249b 249b 249b
green open .internal.alerts-observability.logs.alerts-default-000001          cPrzfK9kTXaxPDiC4JGIPg 1 0 0 0 249b 249b 249b
green open .internal.alerts-observability.uptime.alerts-default-000001        IVfBmW63QiO_rRQIEuhSmQ 1 0 0 0 249b 249b 249b
green open .internal.alerts-ml.anomaly-detection.alerts-default-000001        R5vMIIJYSKWnGUh0ZCY9Xg 1 0 0 0 249b 249b 249b
green open .internal.alerts-observability.slo.alerts-default-000001           dFQ16q2mSWyIexkw86DD6Q 1 0 0 0 249b 249b 249b
green open .internal.alerts-default.alerts-default-000001                     vklvaNfcQiWJW9dSWPLBzQ 1 0 0 0 249b 249b 249b
green open .internal.alerts-observability.apm.alerts-default-000001           v_CAhL2eTEmkcL8I4YVZ7A 1 0 0 0 249b 249b 249b
green open .internal.alerts-observability.metrics.alerts-default-000001       X2yn74wITeOVL606uUgoVA 1 0 0 0 249b 249b 249b
green open .internal.alerts-ml.anomaly-detection-health.alerts-default-000001 nC5SMqeNR4y39oeY8CKNWw 1 0 0 0 249b 249b 249b
green open .internal.alerts-observability.threshold.alerts-default-000001     _QrSVe7VSZyBhNfAQnE3HQ 1 0 0 0 249b 249b 249b
green open .internal.alerts-security.alerts-default-000001                    XPlzMepYQWuNFVf8lDO5lA 1 0 0 0 249b 249b 249b
green open .internal.alerts-stack.alerts-default-000001                       eeEF_u0AQvuKxdfnvo-RZQ 1 0 0 0 249b 249b 249b

Yes, I deleted and created the index, added new documents and everything worked fast, but now for some reason there are problems

I don't see any errors

I have now restarted my pc, re-launched elastic and kibana and it worked fine, although I have done this 3 times before and nothing worked. Very strange behaviorVery strange behavior

Hi @Unknown_Device Welcome to the community

What did you set the JVM to?
Otherwise, by default elasticsearch will take half of the ram on your machine....
Then after while his other programs try to take the ram, it'll mess up elasticsearch
So why don't you try setting elasticsearch jvm to like 4 GB or 8 GB?...

There are a few more methods to search for bottleneck or error:

Since you didn't provide more details, be advised than ELK will use ~ 50% RAM for ES and 2 GB for Kibana by default.

Is it slow, or completely unresponsive?

From what you've described it sounds like the latter - if you access it from your .NET code, it never returns even though it is accessible from your browser.

I'd suggest that you narrow down the differences between the things that work and the things that don't until you know exactly what makes a difference.

  • Are you running the .NET code from the same machine as the browser? (without additional layers like a container, VM or WSL)

  • Can you a very simple piece of .NET code that calls the root (/) endpoint like in your browser example? Make sure it's using exactly the same URL as you used in your browser - same protocol (http vs https), same address, same port, same path.

  • If that's all working, then start to call different endpoints until you can tell what makes it fail. Is it PUT vs GET? A particular index?