# Create Monitor - Define extraction query

**URL:** <https://discuss.elastic.co/t/create-monitor-define-extraction-query/189369>\
**Category:** Kibana\
**Tags:** elastic-stack-alerting\
**Created:** [July 8, 2019, 3:06pm UTC](https://discuss.elastic.co/t/create-monitor-define-extraction-query/189369 "2019-07-08T15:06:37Z")\
**Posts on this page:** 13\
**Page:** 1

<div class="post-metadata">

**Author:** ![ben.sharp](https://avatars.discourse-cdn.com/v4/letter/b/f08c70/32.png) [@ben.sharp](https://discuss.elastic.co/u/ben.sharp)\
**Post date:** [July 8, 2019, 3:06pm UTC](https://discuss.elastic.co/t/create-monitor-define-extraction-query/189369/1 "2019-07-08T15:06:37Z")

</div>

Hello,

I am attempting to create a monitor in Kibana using the "Define using extraction query" option.

However I am struggling to form the query.

I am trying to monitor the last hour of logs, and look for the field "action.keyword" where value = "BLOCK".

From looking at the script that is created from the "Define using visual graph" I believe I have found how to look at the past hour:

```auto
            "filter": [
                {
                    "range": {
                        "timestamp": {
                            "from": "{{period_end}}||-1h",
                            "to": "{{period_end}}",
                            "include_lower": true,
                            "include_upper": true,
                            "format": "epoch_millis",
                            "boost": 1
                        }
                    }
                }
            ],

```

However I am unsure how to then look only at "action.keyword", and then only "BLOCK" values for that.

Could someone please let me know what parameters I should be using here?

Thanks 🙂

---

<div class="post-metadata">

**Author:** ![George\_Marcel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/george_marcel/32/45582_2.png) [@George\_Marcel](https://discuss.elastic.co/u/George_Marcel)\
**Post date:** [July 11, 2019, 2:54pm UTC](https://discuss.elastic.co/t/create-monitor-define-extraction-query/189369/2 "2019-07-11T14:54:30Z")

</div>

Can you try this?

{  
"size": 10,  
"query": {  
"bool": {  
"must": {  
"query\_string": {  
"query": "BLOCK",  
"boost": 1  
}  
},  
"filter": [  
{  
"range": {  
"@timestamp": {  
"from": "{{period\_end}}||-1h",  
"to": "{{period\_end}}",  
"include\_lower": true,  
"include\_upper": true,  
"boost": 1  
}  
}  
}  
]  
}  
}  
}

---

<div class="post-metadata">

**Author:** ![ben.sharp](https://avatars.discourse-cdn.com/v4/letter/b/f08c70/32.png) [@ben.sharp](https://discuss.elastic.co/u/ben.sharp)\
**Post date:** [July 17, 2019, 10:57am UTC](https://discuss.elastic.co/t/create-monitor-define-extraction-query/189369/3 "2019-07-17T10:57:29Z")

</div>

> [@ben.sharp](#):
>
> action.keyword

Hi George, thanks for getting back to me.

I tried running this but got no hits in response:

{  
"\_shards": {  
"total": 10,  
"failed": 0,  
"successful": 10,  
"skipped": 0  
},  
"hits": {  
"hits": ,  
"total": 0,  
"max\_score": null  
},  
"took": 0,  
"timed\_out": false  
}

I generated some logs with "BLOCK" in the field "action.keyword" to ensure that the query was able to find results, but still returned 0 hits unfortunately.

---

<div class="post-metadata">

**Author:** ![George\_Marcel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/george_marcel/32/45582_2.png) [@George\_Marcel](https://discuss.elastic.co/u/George_Marcel)\
**Post date:** [July 17, 2019, 2:38pm UTC](https://discuss.elastic.co/t/create-monitor-define-extraction-query/189369/4 "2019-07-17T14:38:34Z")

</div>

Can you check if you are able to see the data coming up in discover?  
Also check if you are using the right index.

---

<div class="post-metadata">

**Author:** ![ben.sharp](https://avatars.discourse-cdn.com/v4/letter/b/f08c70/32.png) [@ben.sharp](https://discuss.elastic.co/u/ben.sharp)\
**Post date:** [July 18, 2019, 3:47pm UTC](https://discuss.elastic.co/t/create-monitor-define-extraction-query/189369/5 "2019-07-18T15:47:02Z")

</div>

Hi George,

I can see the data in discover, and confirmed I'm using the right index. I'm looking at multiple indexes by using a wildcard, in my case I'm looking at "awswaf-\*" specifically.

---

<div class="post-metadata">

**Author:** ![George\_Marcel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/george_marcel/32/45582_2.png) [@George\_Marcel](https://discuss.elastic.co/u/George_Marcel)\
**Post date:** [July 19, 2019, 1:59pm UTC](https://discuss.elastic.co/t/create-monitor-define-extraction-query/189369/6 "2019-07-19T13:59:57Z")

</div>

Can you paste the query and output from the discover?  
Try increasing the time filter to 24 hrs.  
"from": "{{period\_end}}||-24h",

---

<div class="post-metadata">

**Author:** ![ben.sharp](https://avatars.discourse-cdn.com/v4/letter/b/f08c70/32.png) [@ben.sharp](https://discuss.elastic.co/u/ben.sharp)\
**Post date:** [July 25, 2019, 3:47pm UTC](https://discuss.elastic.co/t/create-monitor-define-extraction-query/189369/7 "2019-07-25T15:47:34Z")

</div>

Hi George, sorry for the delay. Here's the query from the discover:

{  
"version": true,  
"size": 500,  
"sort": [  
{  
"timestamp": {  
"order": "desc",  
"unmapped\_type": "boolean"  
}  
}  
],  
"\_source": {  
"excludes":   
},  
"aggs": {  
"2": {  
"date\_histogram": {  
"field": "timestamp",  
"interval": "30s",  
"time\_zone": "Europe/London",  
"min\_doc\_count": 1  
}  
}  
},  
"stored\_fields": [  
"_"  
],  
"script\_fields": {},  
"docvalue\_fields": [  
{  
"field": "timestamp",  
"format": "date\_time"  
}  
],  
"query": {  
"bool": {  
"must": [  
{  
"query\_string": {  
"query": "action:BLOCK",  
"analyze\_wildcard": true,  
"default\_field": "_"  
}  
},  
{  
"range": {  
"timestamp": {  
"gte": 1564067691755,  
"lte": 1564069491755,  
"format": "epoch\_millis"  
}  
}  
}  
],  
"filter": ,  
"should": ,  
"must\_not":   
}  
},  
"highlight": {  
"pre\_tags": [  
"@kibana-highlighted-field@"  
],  
"post\_tags": [  
"@/kibana-highlighted-field@"  
],  
"fields": {  
"\*": {}  
},  
"fragment\_size": 2147483647  
}  
}

I tried increasing the time filter but that stilled returned 0 hits on the monitor query.

---

<div class="post-metadata">

**Author:** ![George\_Marcel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/george_marcel/32/45582_2.png) [@George\_Marcel](https://discuss.elastic.co/u/George_Marcel)\
**Post date:** [July 25, 2019, 4:24pm UTC](https://discuss.elastic.co/t/create-monitor-define-extraction-query/189369/8 "2019-07-25T16:24:36Z")

</div>

Pasting the same query in the monitor's extraction query should give you the same output as in discover. Can you try that?

---

<div class="post-metadata">

**Author:** ![ben.sharp](https://avatars.discourse-cdn.com/v4/letter/b/f08c70/32.png) [@ben.sharp](https://discuss.elastic.co/u/ben.sharp)\
**Post date:** [July 25, 2019, 4:42pm UTC](https://discuss.elastic.co/t/create-monitor-define-extraction-query/189369/9 "2019-07-25T16:42:51Z")

</div>

I tried copying and pasting straight from one to the other, however the formatting was wrong (some unexpected "}" and "," characters), so I tried to neaten it up a bit, but was then given a "Bad String" error.

I tried cutting it down further to see if I could remove the bad string but was not able to.

Here's what I cut it down to after removing the irrelevant parts (it looks basically the same as the first query you provided):

```
{
"size": 5000,
"_source": {
"query": {
"bool": {
"must": [
{
"query_string": {
"query": "action:BLOCK",
"analyze_wildcard": true,
"default_field": ""
}
},
{
"range": {
"timestamp": {
                            "from": "{{period_end}}||-1h",
                            "to": "{{period_end}}",
                            "include_lower": true,
                            "include_upper": true,
                            "boost": 1
                        }
}
}

```

This has the bad string error on the last line.

Before removing any fields, this is the query I have:

```
{
"version": true,
"size": 500,
"sort": [
{
"timestamp": {
"order": "desc",
"unmapped_type": "boolean"
}
}
],
"_source": {
"excludes": {
},
"aggs": {
"2": {
"date_histogram": {
"field": "timestamp",
"interval": "30s",
"time_zone": "Europe/London",
"min_doc_count": 1
}
}
},
"stored_fields": [
""
],
"script_fields": {},
"docvalue_fields": [
{
"field": "timestamp",
"format": "date_time"
}
],
"query": {
"bool": {
"must": [
{
"query_string": {
"query": "action:BLOCK",
"analyze_wildcard": true,
"default_field": ""
}
},
{
"range": {
"timestamp": {
"gte": 1564067691755,
"lte": 1564069491755,
"format": "epoch_millis"
}
}
}
],
"filter": {},
"should":{} ,
"must_not": {
}
},
"highlight": {
"pre_tags": [
"@kibana-highlighted-field@"
],
"post_tags": [
"@/kibana-highlighted-field@"
],
"fields": {
"*": {}
},
"fragment_size": 2147483647
},
}

```

This still give me a "bad string" error on the last line.

---

<div class="post-metadata">

**Author:** ![George\_Marcel](https://sea2.discourse-cdn.com/elastic/user_avatar/discuss.elastic.co/george_marcel/32/45582_2.png) [@George\_Marcel](https://discuss.elastic.co/u/George_Marcel)\
**Post date:** [July 25, 2019, 5:02pm UTC](https://discuss.elastic.co/t/create-monitor-define-extraction-query/189369/10 "2019-07-25T17:02:50Z")

</div>

Can you try with this query?

{  
"version": true,  
"size": 500,  
"sort": [{  
"timestamp": {  
"order": "desc",  
"unmapped\_type": "boolean"  
}  
}],  
"\_source": {  
"excludes":   
},  
"aggs": {  
"2": {  
"date\_histogram": {  
"field": "timestamp",  
"interval": "30s",  
"time\_zone": "Europe/London",  
"min\_doc\_count": 1  
}  
}  
},  
"stored\_fields": [  
""  
],  
"script\_fields": {},  
"docvalue\_fields": [{  
"field": "timestamp",  
"format": "date\_time"  
}],  
"query": {  
"bool": {  
"must": [{  
"query\_string": {  
"query": "action:BLOCK",  
"analyze\_wildcard": true,  
"default\_field": "_"  
}  
},  
{  
"range": {  
"timestamp": {  
"gte": 1564067691755,  
"lte": 1564069491755,  
"format": "epoch\_millis"  
}  
}  
}  
],  
"filter": [],  
"should": [],  
"must\_not": []  
}  
},  
"highlight": {  
"pre\_tags": [  
"@kibana-highlighted-field@"  
],  
"post\_tags": [  
"@/kibana-highlighted-field@"  
],  
"fields": {  
"_": {}  
},  
"fragment\_size": 2147483647  
}  
}

---

<div class="post-metadata">

**Author:** ![ben.sharp](https://avatars.discourse-cdn.com/v4/letter/b/f08c70/32.png) [@ben.sharp](https://discuss.elastic.co/u/ben.sharp)\
**Post date:** [July 29, 2019, 2:23pm UTC](https://discuss.elastic.co/t/create-monitor-define-extraction-query/189369/11 "2019-07-29T14:23:46Z")

</div>

Tried that, which worked, then modified it to the following:

{  
"version": true,  
"size": 0,  
"sort": [{  
"timestamp": {  
"order": "desc",  
"unmapped\_type": "boolean"  
}  
}],  
"\_source": {  
"excludes":   
},  
"aggs": {  
"2": {  
"date\_histogram": {  
"field": "timestamp",  
"interval": "30s",  
"time\_zone": "Europe/London",  
"min\_doc\_count": 1  
}  
}  
},  
"stored\_fields": [  
""  
],  
"script\_fields": {},  
"docvalue\_fields": [{  
"field": "timestamp",  
"format": "date\_time"  
}],  
"query": {  
"bool": {  
"must": [{  
"query\_string": {  
"query": "action:BLOCK",  
"analyze\_wildcard": true,  
"default\_field": ""  
}  
},  
{  
"range": {  
"timestamp": {  
"from": "{{period\_end}}||-10m",  
"to": "{{period\_end}}",  
"include\_lower": true,  
"include\_upper": true,  
"boost": 1  
}  
}  
}  
],  
"filter": ,  
"should": ,  
"must\_not":   
}  
}  
}

It's now doing what I want it to do.

Thank you for your help George!

---

<div class="post-metadata">

**Author:** ![ben.sharp](https://avatars.discourse-cdn.com/v4/letter/b/f08c70/32.png) [@ben.sharp](https://discuss.elastic.co/u/ben.sharp)\
**Post date:** [July 29, 2019, 2:27pm UTC](https://discuss.elastic.co/t/create-monitor-define-extraction-query/189369/12 "2019-07-29T14:27:06Z")

</div>

In fact, managed to cut it down even further:

{  
"version": true,  
"size": 0,  
"query": {  
"bool": {  
"must": [{  
"query\_string": {  
"query": "action:BLOCK",  
"analyze\_wildcard": true  
}  
},  
{  
"range": {  
"timestamp": {  
"from": "{{period\_end}}||-10m",  
"to": "{{period\_end}}",  
"include\_lower": true,  
"include\_upper": true,  
"boost": 1  
}  
}  
}  
]  
}  
}  
}

---

<div class="post-metadata">

**Author:** ![system](https://us1.discourse-cdn.com/elastic/original/3X/1/a/1ac57faf039f6b580b3f104ef42a2a89e41014de.png) [@system](https://discuss.elastic.co/u/system)\
**Post date:** [August 26, 2019, 2:27pm UTC](https://discuss.elastic.co/t/create-monitor-define-extraction-query/189369/13 "2019-08-26T14:27:10Z")

</div>

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.