No alive nodes found in your cluster

Elastic Stack Elasticsearch
1

I'm on Ubuntu 20.04 with Elasticsearch 7.10.2 (though this has happened in previous minor versions too)

We have a PHP script that runs every night using 7.12.0 of the PHP SDK where it re-indexes live documents and deletes old documents no longer considered live in our CMS. The basic loop is

for each company

  • re-index live stuff
  • delete old stuff

Every so often, maybe every 5-7 days one of the nights we will see an error of "No alive nodes found in your cluster" after trying to do a delete

$deleteCompanyParams = [
  'refresh' => true,
  'index' => "{$company->slug}_{$company->id}",
  'body' => [
    'query' => [
      'terms' => [
        'publication_id' => array_values($oldPublicationIds->toArray())
      ]
    ]
  ]
];
$client->deleteByQuery($deleteCompanyParams);

Any ideas on what can cause this sporadic behavior?

2

Is there anything matching that time in your Elasticsearch master logs that might provide a hint?

3

Hmm, I see I have more digging to do.

[2021-05-05T08:00:00,365][INFO ][o.e.n.Node               ] [prod.v2.elasticsearch] stopping ...
[2021-05-05T08:00:00,551][INFO ][o.e.x.w.WatcherService   ] [prod.v2.elasticsearch] stopping watch service, reason [shutdown initiated]

Looks like unattended-upgrades began then.

Thank you and sorry to clog up the forums needlessly!

4

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.