Painless script fields: How to get date.start date.end for a http_status=!200

Elastic Stack Elasticsearch
1

I want to create a new field that counts the downtime of my website, so if http_status is different then 200; I need a total variable containing the difference between [@timestamp]date.start and [@timestamp]date.end.

So is there any way to get those two dates?

here's my idea:

if (doc['http_status'].value!=200) 
	{ total+=(doc['@timestamp'].date.minuteOfHour.start - doc['@timestamp'].date.minuteOfHour.end);
               return total; 
             }
2

Not sure life is that simple.

404s aren't 200s but don't mean a site is down.
If your site is heading down there'll be slower response times.
If your site is down-down there'll be no logs

You could look at using Watcher to alert on deviations from traffic levels you define as "normal" (see https://www.elastic.co/guide/en/x-pack/current/watcher-getting-started.html)

If you want a solution to automatically learn what is "normal" then see https://www.elastic.co/products/x-pack/machine-learning

3

Thanks for the information, but as I said I want to have a field "total" where I have the difference between [@timestamp]date.start and [@timestamp]date.end - this scripted field is just one step of my idea.

4

Scripted fields are strictly per document, and it looks like you calculation wants to compare timestamps from different events (as each event tends to have a single timestamp). I therefore do not think what you want to do can be done through scripted fields.

5

Alright so what I want to do is having the total variable which is always incremented when we have a http_status =! 200 for a precise URL, so at the end of each month I have this "total" variable containing the time where my website was not reachable.

6

As mark explained, that may be misleading. If you are looking to monitor uptime, why not instead do this through Heartbeat?

7

Yeah I got the point but I don't want to have something running on the website server, so I have a script checking the URI and pushing the following data into elastic:

"properties": {
        "@timestamp": {
          "type": "date",
          "format" : "epoch_second"
        },
        "uri": {
          "type": "keyword"
        },
        "http_status": {
          "type": "integer"
        },
        "response_time": {
          "type": "float"
        }
      }

Now based on these data I wanted to have a new scripted field to get the total I already explained, so if that's not possible with scripted fields is there another solution from this point to get the "total" variable that I want to report?

8

If you know you're running this with a specific periodicity, you could perhaps add a field representing the time in seconds this measurement represent and aggregate across this?

9

Yes indeed I can add this field in my script, it will take some time but it's possible - for now let's consider that I have a new field "failure" which contains in sec the time of each https_status != 200, then a "total_failure" could be scripted with scripted fields for each month or not?

10

No, you would get this through an aggregation query, e.g. through a Kibana dashboard or directly using one of the language clients. Scripted fields are strictly for creating new, calculated fields per document.

11

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.