Watcher for monitor a process does not work

Elastic Stack Kibana
1

We have created a watcher to monitor a process in a linux server but it does not fired and only taking the sleeping state of the process .

{
  "trigger": {
    "schedule": {
      "interval": "5m"
    }
  },
  "input": {
    "search": {
      "request": {
        "search_type": "query_then_fetch",
        "indices": [
          "nonprod-metric_beat"
        ],
        "rest_total_hits_as_int": true,
        "body": {
          "query": {
            "bool": {
              "must": [
                {
                  "range": {
                    "@timestamp": {
                      "gte": "now-5m"
                    }
                  }
                },
                {
                  "terms": {
                    "beat.hostname.keyword": [
                      "cgreupaio05uat"
                    ]
                  }
                },
                {
                  "term": {
                    "application.keyword": "tws"
                  }
                },
                {
                  "terms": {
                    "system.process.name.keyword": [
                      "jobman"
                    ]
                  }
                },
                {
                  "terms": {
                    "system.process.state": [
                      "running"
                    ]
                  }
                }
              ]
            }
          },
          "aggs": {
            "hostwise_agg": {
              "terms": {
                "field": "beat.hostname.keyword"
              },
              "aggs": {
                "service_wise": {
                  "terms": {
                    "field": "system.process.name.keyword"
                  },
                  "aggs": {
                    "service_cmdline": {
                      "terms": {
                        "field": "cmdline.keyword"
                      }
                    }
                  }
                }
              }
            }
          },
          "script_fields": {
            "description": {
              "script": {
                "lang": "painless",
                "source": "params.value",
                "params": {
                  "value": "Service Status Down"
                }
              }
            },
            "criticality": {
              "script": {
                "lang": "painless",
                "source": "params.value",
                "params": {
                  "value": "Warning"
                }
              }
            }
          },
          "_source": []
        }
      }
    }
  },
  "condition": {
    "compare": {
      "ctx.payload.hits.total": {
        "gt": 0
      }
    }
  },
  "actions": {
    "index_payload": {
      "index": {
        "index": "<watcher-alert-index-{now/d}>",
        "doc_type": "my-new-type",
        "execution_time_field": "@timestamp"
      }
    },
    "email_notifcation": {
      "email": {
        "profile": "standard",
        "from": "alerts@canon-europe.com",
        "to": [
          "TCS_TAM_ELK@CANONEUROPENV.onmicrosoft.com"
        ],
        "subject": "TWS | Jobman status: down | P2 | PRD",
        "body": {
          "text": """#account: canontcs
#source: web
#service_instance: TCS-Monitoring-TWS
#category: incident 
#impact: medium 
#ci: prd-tws-process-warning

Hi Team, 
You are receiving this mail because we have found an incident. The below Service Status is down 


{{#ctx.payload.aggregations.hostwise_agg.buckets}} {{key}}: 


{{#service_wise.buckets}} {{key}}: 


 {{#service_cmdline.buckets}} {{key}} {{/service_cmdline.buckets}}


 {{/service_wise.buckets}} {{/ctx.payload.aggregations.hostwise_agg.buckets}}


"""
        }
      }
    }
  }
}
2

Take a look at your watcher history to see what might be going on. For example:

GET .watcher-history-*/_search?size=100
{
  "sort" : [
    { "@timestamp": "desc"}
  ]
}

If nothing stands out there, enabled debug logging for watcher, and see what is in your logs. For example:

PUT /_cluster/settings
{
  "persistent": {
    "org.elasticsearch.xpack.watcher": "DEBUG"
  }
}

Another thing to check is your SMTP server log -- several times I have seen a problem in the SMTP server, and Elasticsearch won't know anything about that.