# Creating a watcher to check for no data in all the indices

**URL:** <https://discuss.elastic.co/t/creating-a-watcher-to-check-for-no-data-in-all-the-indices/306808>\
**Category:** Kibana\
**Tags:** elastic-stack-alerting\
**Created:** [June 9, 2022, 3:03pm UTC](https://discuss.elastic.co/t/creating-a-watcher-to-check-for-no-data-in-all-the-indices/306808 "2022-06-09T15:03:23Z")\
**Posts on this page:** 4\
**Page:** 1

<div class="post-metadata">

**Author:** ![mangeshmj1992](https://avatars.discourse-cdn.com/v4/letter/m/d9b06d/32.png) [@mangeshmj1992](https://discuss.elastic.co/u/mangeshmj1992)\
**Post date:** [June 9, 2022, 3:03pm UTC](https://discuss.elastic.co/t/creating-a-watcher-to-check-for-no-data-in-all-the-indices/306808/1 "2022-06-09T15:03:23Z")

</div>

Hello team,  
I need to create watcher to check for no data in all the indices. I have total 100+ indices.

1. I am using below script. in indices, i am putting \* is this work?
2. I need all index name in body which have 0 records from last 15 min

```auto
{
  "trigger": {
    "schedule": {
      "interval": "240m"
    }
  },
  "input": {
    "search": {
      "request": {
        "search_type": "query_then_fetch",
        "indices": [
          "*"
        ],
        "rest_total_hits_as_int": true,
        "body": {
          "query": {
    "bool": {
      "must": [],
      "filter": [
        {
          "match_all": {}
        },
        {
          "range": {
            "@timestamp": {
              "gte": "now-15m",
              "lte": "now",
              "format": "strict_date_optional_time"
            }
          }
        }
      ],
      "should": [],
      "must_not": []
    }
  }
        }
      }
    }
  },
  "condition": {
    "compare": {
      "ctx.payload.hits.total": {
        "lte": 0
      }
    }
  },
  "actions": {
    "send_email": {
      "email": {
        "profile": "standard",
        "from": "mj@gmail.com",
        "to": [
          "abc@gmail.com"
        ],
        "subject": "Test Email :: There is no log data from last 15 min in below indices",
        "body": {
          "html": """<html>
  <body>
    <strong>There is no log data from last 15 min in below indices </strong>

// need to display list of indices here.

    <br />
 <br /> 
 
  </body>
</html>
"""
        }
      }
    }
  }
}

```

---

<div class="post-metadata">

**Author:** ![richcollier](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/richcollier/32/115035_2.png) [@richcollier](https://discuss.elastic.co/u/richcollier)\
**Post date:** [June 9, 2022, 5:54pm UTC](https://discuss.elastic.co/t/creating-a-watcher-to-check-for-no-data-in-all-the-indices/306808/2 "2022-06-09T17:54:43Z")

</div>

Could to something like this:

```auto
{
    "trigger": {
      "schedule": {
        "interval": "15m"
      }
    },
    "metadata": {
      "longer_time": "14d/d",
      "shorter_time": "15m",
      "alarm_type": "Packet Loss"
    },
    "input": {
      "search": {
        "request": {
          "search_type": "query_then_fetch",
          "indices": [
            "*"
          ],
          "rest_total_hits_as_int": true,
          "body": {
            "size": 0,
            "aggs": {
              "index_names": {
                "terms": {
                  "field": "_index",
                  "size": 10000
                },
                "aggs": {
                  "older_data": {
                    "filter": {
                      "query_string": {
                        "default_field": "@timestamp",
                        "query": "@timestamp:[now-{{ctx.metadata.longer_time}} TO now-{{ctx.metadata.shorter_time}}]"
                      }
                    }
                  },
                  "newer_data": {
                    "filter": {
                      "query_string": {
                        "default_field": "@timestamp",
                        "query": "@timestamp:[now-{{ctx.metadata.shorter_time}} TO now]"
                      }
                    }
                  },
                  "expose_olders_not_in_newers": {
                    "bucket_selector": {
                      "buckets_path": {
                        "older": "older_data._count",
                        "newer": "newer_data._count"
                      },
                      "script": "params.older > 0 && params.newer == 0"
                    }
                  }
                }
              },
              "final_count": {
                "stats_bucket": {
                  "buckets_path": "index_names._count"
                }
              }
            }
          }
        }
      }
    },
    "condition": {
      "script": """
        return ctx.payload.aggregations.final_count.count > 0; 
          """
    },
    "actions": {
      "log": {
        "transform": {
          "script": """
                    return ctx.payload.aggregations.index_names.buckets.stream().map(p -> ['index':p.key,'docs_in_last_week':p.older_data.doc_count,'docs_in_last_15m':p.newer_data.doc_count]).collect(Collectors.toList());
          """
        },
        "logging": {
          "text": """
         {{#ctx.payload._value}}
         index={{index}} had no docs in last 15m (was {{docs_in_last_week}} documents in last 14 days)
         {{/ctx.payload._value}}

"""
        }
      }
    }
  }

```

Sample result is:

```auto
      "actions" : [
        {
          "id" : "log",
          "type" : "logging",
          "status" : "success",
          "transform" : {
            "type" : "script",
            "status" : "success",
            "payload" : {
              "_value" : [
                {
                  "docs_in_last_15m" : 0,
                  "index" : "alert-messages-new",
                  "docs_in_last_week" : 5599
                },
                {
                  "docs_in_last_15m" : 0,
                  "index" : "latest-neid-alerts",
                  "docs_in_last_week" : 39
                },
                {
                  "docs_in_last_15m" : 0,
                  "index" : "devices",
                  "docs_in_last_week" : 2
                }
              ]
            }
          },
          "logging" : {
            "logged_text" : """
         index=alert-messages-new had no docs in last 15m (was 5599 documents in last 14 days)
         index=latest-neid-alerts had no docs in last 15m (was 39 documents in last 14 days)
         index=devices had no docs in last 15m (was 2 documents in last 14 days)

"""
          }
        }
      ]

```

Obviously format to your liking and use email not the simple logging action. Also, you probably won't want to look back as far as I did (14 days - maybe you just do 1h). I had to because I don't really have live data coming into my cluster.

---

<div class="post-metadata">

**Author:** ![mangeshmj1992](https://avatars.discourse-cdn.com/v4/letter/m/d9b06d/32.png) [@mangeshmj1992](https://discuss.elastic.co/u/mangeshmj1992)\
**Post date:** [June 10, 2022, 3:57pm UTC](https://discuss.elastic.co/t/creating-a-watcher-to-check-for-no-data-in-all-the-indices/306808/3 "2022-06-10T15:57:47Z")

</div>

Thank you so much @richcollier its worked for me

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [July 8, 2022, 3:58pm UTC](https://discuss.elastic.co/t/creating-a-watcher-to-check-for-no-data-in-all-the-indices/306808/4 "2022-07-08T15:58:38Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.