I am trying to retrieve the "message" field from my logs for level "ERROR"

[Indiresh_Ds]

Hi there,

I am very new to ELK. My apologies if my doubt seems blunt.

I have several different errors in my logs. I am trying to get the "message" alone of each of these errors as an alert to my mailbox.

I tried using ctx.payload.hits.hits.0._source.message as suggested by "Martijn Van Groningen" in Can't get ctx.payload.hits.hits.index.fields.fieldname to work

But this only returns the message field of one error. But this is not my requirement.

For ex: If I have 7 different errors in my logs, I want to retrieve the "message" field for all these 7 errors. Similarly, if I have 100 logs of level "ERROR", Watcher should send me the "message" field of each of these 100 errors.

Please find below the script that I have written in order to achieve this.

---

{
"trigger": {
"schedule": {
"interval": "10h"
}
},
"input": {
"chain": {
"inputs": [
{
"first": {
"search": {
"request": {
"search_type": "query_then_fetch",
"indices": [
"logs"
],
"types": [],
"body": {
"query": {
"constant_score": {
"filter": {
"bool": {
"filter": [
{
"range": {
"@timestamp": {
"gte": "now-24h"
}
}
}
],
"must": [
{
"match": {
"level": "ERROR"
}
},
{
"bool": {
"must": [
{
"match": {
"host": "some IP address"
}
}
]
}
}
]
}
}
}
}
}
}
}
}
},
"condition": {
"always": {}
},
"actions": {
"send_email": {
"email": {
"profile": "standard",
"to": [
"aa@bb.com"
],
"subject": "Different levels of logs in the specified host",
"body": {
"text": "Please find below the messages of different errors in the specified host \n\n ERROR : {{ctx.payload.first.hits.hits.0._source.message}} \n Please check the logs for further details."
}
}
}
}
}

---

I also have other inputs as a part of the input chain. But I haven't included them because they weren't of any concern. Will I have to use aggregations in order to achieve what I'm trying to achieve?

[spinscale (Alexander Reelsen)]

You can loop through search results like this

{{#ctx.payload.hits.hits}}{{_id}}:{{/ctx.payload.hits.hits}}"

Take your time and check out our examples repo for some more help regarding mustache templating

[Indiresh_Ds]

Hello Alexander,

Thank you so much for your help. It works perfectly now!!

I used the following snippet in order to achieve this:

---

{
"trigger": {
"schedule": {
"interval": "10h"
}
},
"input": {
"chain": {
"inputs": [
{
"first": {
"search": {
"request": {
"search_type": "query_then_fetch",
"indices": [
"logs"
],
"types": [],
"body": {
"query": {
"constant_score": {
"filter": {
"bool": {
"filter": [
{
"range": {
"@timestamp": {
"gte": "now-24h"
}
}
}
],
"must": [
{
"match": {
"level": "ERROR"
}
},
{
"bool": {
"must": [
{
"match": {
"host": "some IP address"
}
}
]
}
}
]
}
}
}
},
"_source": [
"message"
]
}
}
}
}
},
"condition": {
"compare": {
"ctx.payload.first.hits.total": {
"gt": 15
}
}
},
"actions": {
"send_email": {
"email": {
"profile": "standard",
"to": [
"aa@bb.com"
],
"subject": "ERRORS in your specified host are exceeding a count of 15 - Errors that occurred over the past 24 hours",
"body": {
"text": "{{ctx.payload.first.hits.total}} errors have occurred in your specified host \n\n The errors that have occurred over the past 24 hours can be found below:\n\n {{#ctx.payload.first.hits.hits}}{{_source}}:\n{{/ctx.payload.first.hits.hits}}
}
}
}
}
}

---

I also found that I could retrieve other components from my logs by simply appending their tags as comma separated values in the "_source": [ ] block.

For example: "_source" : [ "message", "@timestamp" ] would give me both the messages of the errors in my system and also their corresponding time stamps.

[Indiresh_Ds]

Hello @spinscale ,

Like I mentioned earlier, I was able to retrieve the timestamp as well. But this time that I am seeing in the output is not the same as my local time. I understand that this is due to the timezone difference between UTC and IST. So I used the "time_zone" parameter as documented at https://www.elastic.co/guide/en/elasticsearch/reference/5.5/query-dsl-range-query.html#_time_zone_in_range_queries

---

Please find my script below:

{
"trigger": {
"schedule": {
"interval": "10h"
}
},
"input": {
"chain": {
"inputs": [
{
"first": {
"search": {
"request": {
"search_type": "query_then_fetch",
"indices": [
"logs"
],
"types": [],
"body": {
"query": {
"constant_score": {
"filter": {
"bool": {
"filter": [
{
"range": {
"@timestamp": {
"gte": "now-24h",
"time_zone": "+05:30"
}
}
}
],
"must": [
{
"match": {
"level": "ERROR"
}
},
{
"bool": {
"must": [
{
"match": {
"host": "some IP address"
}
}
]
}
}
]
}
}
}
},
"_source": [
"@timestamp",
"message"
]
}
}
}
}
},
"condition": {
"compare": {
"ctx.payload.first.hits.total": {
"gt": 15
}
}
},
"actions": {
"send_email": {
"email": {
"profile": "standard",
"to": [
"aa@bb.com"
],
"subject": "ERRORS in your specified host are exceeding a count of 15 - Errors that occurred over the past 24 hours",
"body": {
"text": "{{ctx.payload.first.hits.total}} errors have occurred in your specified host \n\n The errors that have occurred over the past 24 hours can be found below:\n\n {{#ctx.payload.first.hits.hits}}{{_source}}:\n{{/ctx.payload.first.hits.hits}}
}
}
}
}
}

---

Is there any way I can get the timestamp in my local time? Because using the "time_zone" parameter is not helping.

[spinscale (Alexander Reelsen)]

the time zone parameter is only applied on query time. If the original time is part of your JSON document, you will be able to access it via the hits array in your payload.

[Indiresh_Ds]

Ummm. I'm afraid I don't quite understand. Can you please elaborate? Is there any link that I can refer?

[Indiresh_Ds]

@spinscale Are you saying that I should write something like {{ctx.payload.hits.hits._source.@timestamp}} ??