Create Monitor - Define extraction query

ben.sharp · July 8, 2019, 3:06pm

Hello,

I am attempting to create a monitor in Kibana using the "Define using extraction query" option.

However I am struggling to form the query.

I am trying to monitor the last hour of logs, and look for the field "action.keyword" where value = "BLOCK".

From looking at the script that is created from the "Define using visual graph" I believe I have found how to look at the past hour:

            "filter": [
                {
                    "range": {
                        "timestamp": {
                            "from": "{{period_end}}||-1h",
                            "to": "{{period_end}}",
                            "include_lower": true,
                            "include_upper": true,
                            "format": "epoch_millis",
                            "boost": 1
                        }
                    }
                }
            ],

However I am unsure how to then look only at "action.keyword", and then only "BLOCK" values for that.

Could someone please let me know what parameters I should be using here?

Thanks

George_Marcel · July 11, 2019, 2:54pm

Can you try this?

{
"size": 10,
"query": {
"bool": {
"must": {
"query_string": {
"query": "BLOCK",
"boost": 1
}
},
"filter": [
{
"range": {
"@timestamp": {
"from": "{{period_end}}||-1h",
"to": "{{period_end}}",
"include_lower": true,
"include_upper": true,
"boost": 1
}
}
}
]
}
}
}

ben.sharp · July 17, 2019, 10:57am

Hi George, thanks for getting back to me.

I tried running this but got no hits in response:

{
"_shards": {
"total": 10,
"failed": 0,
"successful": 10,
"skipped": 0
},
"hits": {
"hits": ,
"total": 0,
"max_score": null
},
"took": 0,
"timed_out": false
}

I generated some logs with "BLOCK" in the field "action.keyword" to ensure that the query was able to find results, but still returned 0 hits unfortunately.

George_Marcel · July 17, 2019, 2:38pm

Can you check if you are able to see the data coming up in discover?
Also check if you are using the right index.

ben.sharp · July 18, 2019, 3:47pm

Hi George,

I can see the data in discover, and confirmed I'm using the right index. I'm looking at multiple indexes by using a wildcard, in my case I'm looking at "awswaf-*" specifically.

George_Marcel · July 19, 2019, 1:59pm

Can you paste the query and output from the discover?
Try increasing the time filter to 24 hrs.
"from": "{{period_end}}||-24h",

ben.sharp · July 25, 2019, 3:47pm

Hi George, sorry for the delay. Here's the query from the discover:

{
"version": true,
"size": 500,
"sort": [
{
"timestamp": {
"order": "desc",
"unmapped_type": "boolean"
}
}
],
"_source": {
"excludes":
},
"aggs": {
"2": {
"date_histogram": {
"field": "timestamp",
"interval": "30s",
"time_zone": "Europe/London",
"min_doc_count": 1
}
}
},
"stored_fields": [
""
],
"script_fields": {},
"docvalue_fields": [
{
"field": "timestamp",
"format": "date_time"
}
],
"query": {
"bool": {
"must": [
{
"query_string": {
"query": "action:BLOCK",
"analyze_wildcard": true,
"default_field": ""
}
},
{
"range": {
"timestamp": {
"gte": 1564067691755,
"lte": 1564069491755,
"format": "epoch_millis"
}
}
}
],
"filter": ,
"should": ,
"must_not":
}
},
"highlight": {
"pre_tags": [
"@kibana-highlighted-field@"
],
"post_tags": [
"@/kibana-highlighted-field@"
],
"fields": {
"*": {}
},
"fragment_size": 2147483647
}
}

I tried increasing the time filter but that stilled returned 0 hits on the monitor query.

George_Marcel · July 25, 2019, 4:24pm

Pasting the same query in the monitor's extraction query should give you the same output as in discover. Can you try that?

ben.sharp · July 25, 2019, 4:42pm

I tried copying and pasting straight from one to the other, however the formatting was wrong (some unexpected "}" and "," characters), so I tried to neaten it up a bit, but was then given a "Bad String" error.

I tried cutting it down further to see if I could remove the bad string but was not able to.

Here's what I cut it down to after removing the irrelevant parts (it looks basically the same as the first query you provided):

{
"size": 5000,
"_source": {
"query": {
"bool": {
"must": [
{
"query_string": {
"query": "action:BLOCK",
"analyze_wildcard": true,
"default_field": ""
}
},
{
"range": {
"timestamp": {
                            "from": "{{period_end}}||-1h",
                            "to": "{{period_end}}",
                            "include_lower": true,
                            "include_upper": true,
                            "boost": 1
                        }
}
}

This has the bad string error on the last line.

Before removing any fields, this is the query I have:

{
"version": true,
"size": 500,
"sort": [
{
"timestamp": {
"order": "desc",
"unmapped_type": "boolean"
}
}
],
"_source": {
"excludes": {
},
"aggs": {
"2": {
"date_histogram": {
"field": "timestamp",
"interval": "30s",
"time_zone": "Europe/London",
"min_doc_count": 1
}
}
},
"stored_fields": [
""
],
"script_fields": {},
"docvalue_fields": [
{
"field": "timestamp",
"format": "date_time"
}
],
"query": {
"bool": {
"must": [
{
"query_string": {
"query": "action:BLOCK",
"analyze_wildcard": true,
"default_field": ""
}
},
{
"range": {
"timestamp": {
"gte": 1564067691755,
"lte": 1564069491755,
"format": "epoch_millis"
}
}
}
],
"filter": {},
"should":{} ,
"must_not": {
}
},
"highlight": {
"pre_tags": [
"@kibana-highlighted-field@"
],
"post_tags": [
"@/kibana-highlighted-field@"
],
"fields": {
"*": {}
},
"fragment_size": 2147483647
},
}

This still give me a "bad string" error on the last line.

George_Marcel · July 25, 2019, 5:02pm

Can you try with this query?

{
"version": true,
"size": 500,
"sort": [{
"timestamp": {
"order": "desc",
"unmapped_type": "boolean"
}
}],
"_source": {
"excludes":
},
"aggs": {
"2": {
"date_histogram": {
"field": "timestamp",
"interval": "30s",
"time_zone": "Europe/London",
"min_doc_count": 1
}
}
},
"stored_fields": [
""
],
"script_fields": {},
"docvalue_fields": [{
"field": "timestamp",
"format": "date_time"
}],
"query": {
"bool": {
"must": [{
"query_string": {
"query": "action:BLOCK",
"analyze_wildcard": true,
"default_field": ""
}
},
{
"range": {
"timestamp": {
"gte": 1564067691755,
"lte": 1564069491755,
"format": "epoch_millis"
}
}
}
],
"filter": [],
"should": [],
"must_not": []
}
},
"highlight": {
"pre_tags": [
"@kibana-highlighted-field@"
],
"post_tags": [
"@/kibana-highlighted-field@"
],
"fields": {
"": {}
},
"fragment_size": 2147483647
}
}

ben.sharp · July 29, 2019, 2:23pm

Tried that, which worked, then modified it to the following:

{
"version": true,
"size": 0,
"sort": [{
"timestamp": {
"order": "desc",
"unmapped_type": "boolean"
}
}],
"_source": {
"excludes":
},
"aggs": {
"2": {
"date_histogram": {
"field": "timestamp",
"interval": "30s",
"time_zone": "Europe/London",
"min_doc_count": 1
}
}
},
"stored_fields": [
""
],
"script_fields": {},
"docvalue_fields": [{
"field": "timestamp",
"format": "date_time"
}],
"query": {
"bool": {
"must": [{
"query_string": {
"query": "action:BLOCK",
"analyze_wildcard": true,
"default_field": ""
}
},
{
"range": {
"timestamp": {
"from": "{{period_end}}||-10m",
"to": "{{period_end}}",
"include_lower": true,
"include_upper": true,
"boost": 1
}
}
}
],
"filter": ,
"should": ,
"must_not":
}
}
}

It's now doing what I want it to do.

Thank you for your help George!

ben.sharp · July 29, 2019, 2:27pm

In fact, managed to cut it down even further:

{
"version": true,
"size": 0,
"query": {
"bool": {
"must": [{
"query_string": {
"query": "action:BLOCK",
"analyze_wildcard": true
}
},
{
"range": {
"timestamp": {
"from": "{{period_end}}||-10m",
"to": "{{period_end}}",
"include_lower": true,
"include_upper": true,
"boost": 1
}
}
}
]
}
}
}

system · August 26, 2019, 2:27pm

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.