Logs not visible in kibana - Could not index event to Elasticsearch. 400

Elastic Stack Elasticsearch
1

Dears,

I've just saw that logs aren't comming anymore from beats to kibana since 1 month.
In the syslog, I saw the following warning:

Could not index event to Elasticsearch. 400 Validation Failed: 1: this action would add [1] total shards, but this cluster currently has [1000]/[1000] maximum shards open

Here's some usefull info:
GET / :

{
  "name" : "my.server.com",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "9LEV6xSMSASdG9LKTrZ6CA",
  "version" : {
    "number" : "7.8.1",
    "build_flavor" : "default",
    "build_type" : "rpm",
    "build_hash" : "b5ca9c58fb664ca8bf9e4057fc229b3396bf3a89",
    "build_date" : "2020-07-21T16:40:44.668009Z",
    "build_snapshot" : false,
    "lucene_version" : "8.5.1",
    "minimum_wire_compatibility_version" : "6.8.0",
    "minimum_index_compatibility_version" : "6.0.0-beta1"
  },
  "tagline" : "You Know, for Search"
}

GET /_cat/nodes?v :

ip            heap.percent ram.percent cpu load_1m load_5m load_15m node.role master name
172.16.1.124           55          99   2    0.08    0.10     0.09 dilmrt    *      my.server.com

GET /_cat/health?v :

epoch      timestamp cluster       status node.total node.data shards  pri relo init unassign pending_tasks max_task_wait_time active_shards_percent
1609923445 08:57:25  elasticsearch green           1         1   1000 1000    0    0        0             0                  -                100.0%

GET /_cat/indices?v :

health status index                                            uuid                   pri rep docs.count docs.deleted store.size pri.store.size
green  open   mycompany-lr-preprod-nginx-access-2020.11.20    DpRdSHfRRoC-jGIH7vtPmg   1   0       1452            0    683.2kb        683.2kb
green  open   mycompany-lr-preprod-nginx-access-2020.11.22    6uSqnkdnTBui0A9WXkqM4A   1   0       4149            0      1.3mb          1.3mb
green  open   mycompany-lr-preprod-nginx-access-2020.11.21    meNa8rB0S9uZ4RYezdvZvw   1   0       4397            0      1.5mb          1.5mb
green  open   mycompany-tcs-prod-nginx-error-2020.12.09    EkI7qXBwRaGXomty5s8k7w   1   0   30717381            0      6.4gb          6.4gb
green  open   mycompany-tcs-prod-nginx-error-2020.12.08    Wfwb8jOBTTGtndRc_TA8dg   1   0   29852183            0      6.2gb          6.2gb
green  open   mycompany-tcs-prod-nginx-error-2020.12.07    LmO-sT1iTTahBS8HeATK_Q   1   0   32358745            0      6.8gb          6.8gb
green  open   mycompany-tcs-prod-nginx-error-2020.12.06    cHwRRuOZRkWu1fbx5cWwAQ   1   0    2534894            0    551.8mb        551.8mb
green  open   mycompany-lr-preprod-nginx-error-2020.11.09     TZTh2pNoR_SGS9EKD0lBbQ   1   0          2            0       54kb           54kb
green  open   mycompany-tcs-prod-nginx-error-2020.12.01    DbbXrwE9QxmPBcICFXW1qg   1   0   27926900            0      5.8gb          5.8gb
...
...
...
green  open   preprod-a-cmp-log-2020.11.12             UN-MusO_TtiRUcmaZuASSg   1   0       1524            0        1mb            1mb
green  open   preprod-a-cmp-log-2020.11.13             r1hnR9k4ToWaX9yDBkT29A   1   0      21560            0      5.6mb          5.6mb
green  open   mycompany-lr-preprod-nginx-error-2020.12.30     8VBHGprASgmp1iolCghejg   1   0          5            0    134.9kb        134.9kb

I've removed some of the lines above (see ...) because the list was too long, and it was exceeding the number of accepted lines.
I'm not sure why we have no logs in 2021.
Can you help, please?

2

You just have too many shards per node.
You have very small shards so you should change that I guess.
You also have preprod prod indices. What about removing the preprod ones?
Do you want to keep all your indices around? If not, your can think of using ILM (Index Lifecycle Management) to automatically remove old indices.

Or you can also increase the number of nodes.

3

Thanks for reply @dadoonet

The preprod logs are needed too. Can't remove them.
I will research to see how can I change the shards. They should be bigger?

4

You can have around 20gb to 50gb par shard.
Something like 20 shards per gb of HEAP.

How much heap do you have?

5

If it's about elasticsearch, than the heap is set to 4GB:

-Xms4g
-Xmx4g
6

So you should limit to 80/100 shards on that node IMO.

May I suggest you look at the following resources about sizing:

And Using Rally to Get Your Elasticsearch Cluster Size Right | Elastic Videos

7

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.