Monitoring shars count by watchers

Elastic Stack Kibana
1

Hi all,

Is it possible to monitor the number of shards per node using watchers?

I would like to be able to trigger an alert when the result of the GET _cat/allocation?v&h=node,shards returns, for at least one node, a value greater than 500.

Thank you for your help.

Cristina Marletta Livi

2

Hello @Cristina_Marletta_Li

Yes, it is possible using watcher. Below is just an example, you might have to update as per your requirement.

{
  "trigger": {
    "schedule": {
      "interval": "1m"
    }
  },
  "input": {
    "http": {
      "request": {
        "scheme": "https",
        "host": "elk-end-point",
        "port": 9243,
        "method": "get",
        "path": "/_cat/allocation",
        "params": {
          "format": "json",
          "h": "node,shards"
        },
        "headers": {},
        "auth": {
          "basic": {
            "username": "user",
            "password": "password"
          }
        }
      }
    }
  },
  "condition": {
    "script": {
      "source": """
        for (def entry : ctx.payload.data) {
          if (Integer.parseInt(entry.shards) > 500) {
            return true;
          }
        }
        return false;
      """,
      "lang": "painless"
    }
  },
  "actions": {
    "log_alert": {
      "logging": {
        "level": "info",
        "text": "One or more nodes have more than 500 shards."
      }
    }
  }
}

Thanks!!

3

Thanks @Tortoise . I wonder: are there other, more advisable ways to monitor the number of shards via an alerting mechanism? A rule, for example.

4

Hi @Tortoise ,

I tried your script.

Running 'Simulate', I get a certificate error: (certificate_unknown) PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target

How can I get over it?

Thanks for your help

Cristina

5

Hello @Cristina_Marletta_Li

Other way is you can try to use synthetics in Kibana to monitor the URL & the output we capture is as below :

image
image1074×147 14.1 KB

On the synthetics response you can use the regex to see if output of shards is greater > 499 , on this automatically the synthetics will trigger alert :

Response body contains : 
"shards":"(5\d{2}|[6-9]\d{2}|\d{4,})"

If you still plan to use Watcher please share what watcher you are using for which the error was shared?

Thanks!!

6

Hi Tortoise,

this is my watcher definition:

{
  "trigger": {
    "schedule": {
      "interval": "12h"
    }
  },
  "input": {
    "http": {
      "request": {
        "scheme": "https",
        "host": "127.0.0.1",
        "port": 9200,
        "method": "get",
        "path": "/_cat/allocation",
        "params": {
          "h": "node,shards",
          "format": "json"
        },
        "headers": {}
      }
    }
  },
  "condition": {
    "script": {
      "source": "for (def entry : ctx.payload.data) {if (Integer.parseInt(entry.shards) > 500) {return true;}} return false;",
      "lang": "painless"
    }
  },
  "actions": {
    "log_alert": {
      "logging": {
        "level": "info",
        "text": "One or more nodes have more than 500 shards."
      }
    }
  }
}

the simulation returns:

{
            "type": "s_s_l_handshake_exception",
            "reason": "(certificate_unknown) PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target"
}

The Elasticsearch configuration contains:
xpack.security.http.ssl:
enabled: true
keystore.path: certs/http.p12
certificate_authorities: certs/http_ca.crt

xpack.security.transport.ssl:
enabled: true
verification_mode: certificate
keystore.path: certs/transport.p12
truststore.path: certs/transport.p12

The curl test is ok:
curl https://127.0.0.1:9200 (external, opens in a new tab or window) -v --cacert /etc/elasticsearch/certs/http_ca.crt Trying 127.0.0.1:9200... Connected to 127.0.0.1 (127.0.0.1) port 9200 (#0)

So, I don't understand where the mistake lies.

7

Hello @Cristina_Marletta_Li

As per the post :

In the elasticsearch.yml we need to add below entry :

If it is for testing environment can use below (not secure):
xpack.http.ssl.verification_mode: none

For Prod need to use below :
xpack.http.ssl.certificate_authorities: ["/path/to/http_ca.crt"]

Documentation :

https://www.elastic.co/docs/reference/elasticsearch/configuration-reference/watcher-settings

Thanks!!

8

Hi @Tortoise ,

now it’all ok!

Tnx so much.

Just one more question: On which types of nodes should I configure watchers: hot nodes, masters, etc.? For example, email accounts and xpack.http.ssl.certificate_authorities in elasticsearch.yml.