Healthcheck based on recent log entries

[ag_joeb (Joseph Borg)]

I'm looking to visualize the following in one my my dashboards.

The healthcheck of one of my services would be based on the amount of log entries in the past few seconds; i.e. if there were more than 5 new log entries (in an existing index) in the past 5 seconds, the dashboard's lens shows "TRUE" else shows "FALSE".

Would appreciate if someone can point me in the right direction.

thanks in advance.

[wylie (Wylie Conlon)]

Hi, this is not something that Lens supports today. You need to use a tool that is more programmable, and specifically I think that if you use Vega in Kibana you can achieve this.

[ag_joeb (Joseph Borg)]

Thanks for your reply @wylie Feels like overcomplicating something which should be trivial. Is there perhaps another beat that can be used to check the healthcheck of something based on the activity of a log file ?

[wylie (Wylie Conlon)]

In order to build a visualization, you would need to insert data into a new index representing your status. This might be possible using Kibana's built-in alerting capabilities, but be aware that a 5-second interval is potentially going to slow down Kibana unless you follow the instructions for running in production. You could also set up an external monitoring system that is able to index new logs into Elasticsearch, which you could then visualize in Kibana.

Here are the instructions for setting up a Kibana alert. You could use an index threshold alert, which sends output to an index. Alerting | Kibana Guide [7.13] | Elastic

[ag_joeb (Joseph Borg)]

Thanks for your reply @wylie

I've managed to create a new index containing the string "HEALTHY" or "UNHEALTHY" based on the result of a search query using the watcher API. The part I'm struggling with is including the value of one of the fields to the new index (the agent.name field) to the new index along the "HEALTHY" or "UNHEALTHY" string.

Below is my painless script in the transform argument if it helps:

HashMap result = new HashMap(); result.status = \"Healthy\";result.test_field = ctx._source.agent.name ;if (ctx.payload.hits.total < 3) {result.status = \"Unhealthy\";} return result;

getting the error:

cannot access method/field [agent] from a null def reference

At the following position:

          "result.test_field = ctx._source.agent.name ;",
          "                               ^---- HERE"

What can I do to correct this ?

Thanks in advance

[wylie (Wylie Conlon)]

I don't know enough Painless to solve your problem, but whenever I have issues like that I will try every possible variation of array access patterns. For example, have you tried ctx['_source']['agent.name']? Have you checked for null values in the appropriate places?

If none of that works, you can try painless debugging or asking in the Elasticsearch forums.