Watcher: dynamic index name does not work

Elastic Stack Elasticsearch
1

I'm wanna limit searching indices in my watch by follow https://www.elastic.co/guide/en/elasticsearch/reference/current/date-math-index-names.html, but can not make it work.
Here is my watch indices config:

"indices": ["<my-name-{now/M{YYYY.MM.dd}}>"]

What is the right way?

2

Hey,

can you share the full watch plus the output of the execute watch API for further debugging?

Thanks!

--Alex

3

I'm sorry for being late in response. Here are my watcher configuration and its output:

Watcher configuration:
{ "trigger": { "schedule": { "cron": "0 0/1 * * * ?" } }, "input": { "search": { "request": { "search_type": "query_then_fetch", "indices": [ "<my-name-{now/M{YYYY.MM.dd}}>" ], "types": [], "body": { "query": { "bool": { "filter": [ { "match": { "my-field": "TEXT_MATCH" } }, { "range": { "@timestamp": { "from": "{{ctx.trigger.scheduled_time}}||-60s", "to": "{{ctx.trigger.triggered_time}}" } } } ] } }, "aggs": { "ip_aggs": { "terms": { "field": "ip" } }, "xxx_aggs": { "terms": { "field": "xxx" } } }, "_source": [ "ip", "name", "xxx" ] } } } }, "condition": { "compare": { "ctx.payload.hits.total": { "gte": 1 } } }, "actions": { "my_webhook": { "webhook": { "scheme": "http", "host": "domain.com", "port": 80, "method": "post", "path": "/index.php", "params": {}, "headers": {}, "body": "{{ctx.watch_id}}+++{{#ctx.payload.aggregations.src_ip_aggs.buckets}} {{key}}({{doc_count}}) {{/ctx.payload.aggregations.src_ip_aggs.buckets}}^newline^{{#ctx.payload.aggregations.switch_ip_aggs.buckets}} {{key}}({{doc_count}}) {{/ctx.payload.aggregations.switch_ip_aggs.buckets}}" } } }, "throttle_period_in_millis": 180000 }

and watcher output:
{ "watch_id": "my-watch-name", "state": "execution_not_needed", "_status": { "state": { "active": true, "timestamp": "2017-05-29T09:35:11.142Z" }, "last_checked": "2017-05-30T02:29:00.401Z", "actions": { "my_webhook": { "ack": { "timestamp": "2017-05-29T09:35:11.142Z", "state": "awaits_successful_execution" } } } }, "trigger_event": { "type": "schedule", "triggered_time": "2017-05-30T02:29:00.401Z", "schedule": { "scheduled_time": "2017-05-30T02:29:00.000Z" } }, "input": { "search": { "request": { "search_type": "query_then_fetch", "indices": [ "<my-name-{now/M{YYYY.MM.dd}}>" ], "types": [], "body": { "query": { "bool": { "filter": [ { "match": { "my-field": "TEXT_MATCH" } }, { "range": { "@timestamp": { "from": "{{ctx.trigger.scheduled_time}}||-60s", "to": "{{ctx.trigger.triggered_time}}" } } } ] } }, "aggs": { "ip_aggs": { "terms": { "field": "ip" } }, "xxx_aggs": { "terms": { "field": "xxx" } } }, "_source": [ "ip", "name", "xxx" ] } } } }, "condition": { "compare": { "ctx.payload.hits.total": { "gte": 1 } } }, "metadata": { "name": "my-watch-name" }, "result": { "execution_time": "2017-05-30T02:29:00.401Z", "execution_duration": 1, "input": { "type": "search", "status": "success", "payload": { "_shards": { "total": 0, "failed": 0, "successful": 0 }, "hits": { "hits": [], "total": 0, "max_score": 0 }, "took": 0, "timed_out": false }, "search": { "request": { "search_type": "query_then_fetch", "indices": [ "<my-name-{now/M{YYYY.MM.dd}}>" ], "types": [], "body": { "query": { "bool": { "filter": [ { "match": { "my-field": "TEXT_MATCH" } }, { "range": { "@timestamp": { "from": "2017-05-30T02:29:00.000Z||-60s", "to": "2017-05-30T02:29:00.401Z" } } } ] } }, "aggs": { "ip_aggs": { "terms": { "field": "ip" } }, "xxx_aggs": { "terms": { "field": "xxx" } } }, "_source": [ "ip", "name", "xxx" ] } } } }, "condition": { "type": "compare", "status": "success", "met": false, "compare": { "resolved_values": { "ctx.payload.hits.total": 0 } } }, "actions": [] }, "messages": [] }

4

It seems, that the index you are trying to query does not exist.

I think the index should be my-name-2017.05.01. You can try with curl like this, to test if the resolution is working, before putting this into a watch (sorry for the URL encoding)

curl -v 'http://localhost:9200/%3Cmy-name-%7Bnow%2FM%7BYYYY.MM.dd%7D%7D%3E' -u elastic:changeme

{"error":{"root_cause":[{"type":"index_not_found_exception","reason":"no such index","index_uuid":"_na_","index":"my-name-2017.05.01"}],"type":"index_not_found_exception","reason":"no such index","index_uuid":"_na_","index":"my-name-2017.05.01"},"status":404}

--Alex

5

I expected to see my-name-2017.05.01 in the output, instead of the original configuration. The output makes me think that the syntax was invalid. I think better output is my-name-2017.05.01 whether the index does exist or not.

Actually, I index data by week number, for example my-name-2017.22 for current week. I found the %W for the week number, is that right? What's the right configuration for my weekly index in watcher?

Very thank you.

6

What is returned, if you call this without a watch, by just executing a regular search request? Let's try to remove as much out of the equation as possible first.

--Alex

7

Finally, I change my index name from weekly to daily because I can not find the suitable date configure for that. I'm using this index name: <my-name-{now/d{YYYY.MM.dd}}>, it works.

Due to the output, while watcher is working fine, it's still <my-name-{now/d{YYYY.MM.dd}}> instead of my-name-2017.05.31. As I said above, this output confuses me so much and I expected that is my-name-2017.05.31

Thank you for your help.

8

This topic was automatically closed 28 days after the last reply. New replies are no longer allowed.