X-pack Watcher email alert with full information stripped from JSON file

[Jay.Dhillon]

How can define Watcher watches script to detect a keyword from JSON files? and How to include more information from JSON file(From Azure EventHUB)? Currently using below Advanced JSON watch but seems like not working.

{
"trigger": {
"schedule": {
"interval": "1m"
}
},
"input": {
"search": {
"request": {
"search_type": "query_then_fetch",
"indices": [
"eventhub"
],
"types": [],
"body": {
"query": {
"bool": {
"must": [
{
"query_string": {
"query": "Microsoft.Authorization/roleAssignments"
}
},
{
"range": {
"@timestamp": {
"gte": "now-1m"
}
}
}
]
}
},
"_source": [
"message"
],
"sort": [
{
"@timestamp": {
"order": "desc"
}
}
]
}
}
}
},
"condition": {
"compare": {
"ctx.payload.hits.total": {
"gt": 0
}
}
},
"actions": {
"email_me": {
"throttle_period_in_millis": 600000,
"email": {
"profile": "standard",
"to": [
""
],
"subject": "ELK Watcher: AzureRoleAssignmentsChangesDetected",
"body": {
"html": "Watcher detected {{ctx.payload.hits.total}} AzureRoleAssignmentsChanges"
}
}
}
},
"throttle_period_in_millis": 900000
}

[Jay.Dhillon]

I was able to attach JSON file with the each trap but it's including full list of traps/log received by elastic. Is there a filter only to include only log of current time?

"LogEncounter": {
"data": {
"format": "json"
}

[spinscale (Alexander Reelsen)]

Hey,

first, it would be great, if you could properly format code/watcher snippets. As this forum supports github markdown, it should be pretty easy and makes reading watches inifnitely simpler.

Second, you could use the extract directive from the search input to reduce the size of your payload. Alternatively you could use a transform in your action.

--Alex

[Jay.Dhillon]

It seems like it's not working still? Is this correct format? search seems like not valid parameter for action.

"actions": {
"email_me": {
"throttle_period_in_millis": 600000,
"email": {
"profile": "standard",
"priority": "high",
"to": [
"email"
],
"subject": "Detected {{ctx.payload.hits.total}} Azure Role Assignments Changes",
"body": {
"text": "Check Attached for more details"
},
"attachments": {
"pdf.pdf": {
"reporting": {
"url": "****",
"retries":6,
"interval":"1s",
"auth": {
"basic": {
"username": "elastic",
"password": ""
}
}
}
},
"Raw Data": {
"data": {
"format": "json"
}
}
}
}
},
"transform": {
"search": {
"request": {
"search_type": "query_then_fetch",
"indices": [
"eventhub"
],
"types": [],
"body": {
"query": {
"match": {
"field": "string"
},
"range": {
"@timestamp": {
"gte": "now-2m"
}
}
}
}
}
}
}
}

[Jay.Dhillon]

I have specified range inside input but data field inside action still doesn't attach matched events Raw Data. Need to attach JSON(RAW) file with every event trapped.

"input": {
"search": {
"request": {
"search_type": "query_then_fetch",
"indices": [
"eventhub"
],
"types": [],
"body": {
"size": 0,
"query": {
"bool": {
"must": [
{
"match": {
"field": "string"
}
},
{
"range": {
"@timestamp": {
"gte": "now-2m"
}
}
}
]
}
}
}
}
}
}

[spinscale (Alexander Reelsen)]

if you use a transform, you might want to use a script transform and decide manually which data to keep and which to remove. A search transform just uses the search result as new input, so I dont think you will gain anything from that.

I have not looked any closer at the snippets, because they are nearly impossible to read without formatting...